Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,27 @@
through the refusal; the runner transport never resends it. Interactions that mutate without a
leading read (`press`, a coordinate `fill`, `swipe`, `scroll`, hardware keys) keep the foreground
repair and still bring a stopped app up.
- Fixed (ios): a local Simulator snapshot taken through the host AX bridge no longer publishes the
screens a modal presentation covers, so a selector resolves the control the user can reach instead
of the stale copy underneath it. UIKit appends each presentation as a later sibling of what it
presents over, and the bridge reports every sibling, so a React Navigation stack that presents its
pushed routes as sheets kept all three screens live: one capture of a three-screen stack published
567 nodes across three screens where the same app under the XCTest runner published 76 across the
top one. Maestro's visibility predicate is geometric, so `- tapOn: "Push article"` matched the
covered screen's button first and the flow failed on an assertion about the screen that never
arrived, and `presentation: "formSheet"` fields matched intermittently (#2638). Geometry cannot
close this: a sheet leaves most of the covered screen uncovered, and the occlusion pass only
treats floating chrome as a cover. The presentation cut now applies modal containment — when the
last transition view under a container carries UIKit's dimming view as its direct child and that
dimming view takes touches, the earlier transition views that dimmed area spans, and everything
under them, are cut from the regular and interactive snapshots. The bridge now reads
`userInteractionEnabled` on dimming views (source version v1.7.0) because a sheet resting at an
undimmed detent keeps its dimming view while the screen under it stays reachable; that sheet, and
any presentation UIKit does not dim, is left as it was, so the cut asserts containment only where
the producer states it. `snapshot --raw` still reports the covered screens, and producers that
report no UIKit class names — the XCTest runner, whose own queries answered 76 nodes for that same
state, plus `appium-source` and `limrun-ios-tree` — never trigger the cut. All 39 flows of React Navigation's Maestro suite pass on an iPhone 17 Simulator running
iOS 26.2 with this change, including two that never passed on the bridge.
- Fixed (ios): a local Simulator snapshot taken through the host AX bridge once again publishes the
geometric `hittable` fact, so `is hittable` and a `hittable:` selector resolve the same controls on
the bridge and the XCTest runner. The snapshot capability table has declared `hittable =
Expand Down
48 changes: 47 additions & 1 deletion apple/snapshot-bridge/SnapshotBridgeRuntime.m
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
NSString *const kProtocolVersionKey = @"protocolVersion";
NSString *const kSourceVersionKey = @"sourceVersion";
NSString *const kRequestIdKey = @"requestId";
NSString *const kSourceVersion = @"agent-device-simulator-ax-v1.6.0";
NSString *const kSourceVersion = @"agent-device-simulator-ax-v1.7.0";
const NSUInteger kProtocolVersion = 1;
const uint32_t kMaximumFrameBytes = 16 * 1024 * 1024;
const NSUInteger kMaximumDepth = 128;
Expand All @@ -34,8 +34,11 @@
static NSString *const kAttributeAutomationType = @"XC_kAXXCAttributeAutomationType";
static NSString *const kAttributeTraits = @"XC_kAXXCAttributeTraits";
static NSString *const kAttributeChildren = @"XC_kAXXCAttributeChildren";
static NSString *const kAttributeIsUserInteractionEnabled = @"XC_kAXXCAttributeIsUserInteractionEnabled";
static NSString *const kPresentationDimmingViewClass = @"UIDimmingView";
static NSString *const kSnapshotAttributes = @"UIAccessibilitySnapshotKeyAttributes";
static NSString *const kSnapshotChildren = @"UIAccessibilitySnapshotKeyChildren";
static NSString *const kSnapshotElement = @"UIAccessibilitySnapshotKeyElement";
static NSString *const kXctAutomationSupportPath =
@"/Developer/Library/PrivateFrameworks/XCTAutomationSupport.framework/XCTAutomationSupport";
static NSString *const kAxRuntimePath =
Expand Down Expand Up @@ -202,8 +205,35 @@ - (nullable id)jsonValue:(id)value name:(NSString *)name
return nil;
}

/*
* Whether a presentation's dimming view takes touches is what separates a sheet that blocks the
* content under it from one resting at an undimmed detent, and nothing else in the tree says so.
* It is read for dimming views alone: requesting it for every node costs about half again the
* capture time.
*/
- (nullable NSNumber *)userInteractionEnabledForSnapshot:(NSDictionary *)snapshot
options:(nullable NSDictionary *)options
{
id element = snapshot[kSnapshotElement];
NSNumber *attribute = [options[@"attributes"] firstObject];
if (!element || ![attribute isKindOfClass:NSNumber.class]) return nil;
@try {
NSError *failure = nil;
NSDictionary *read = [_framework userTestingSnapshotForElement:element options:options error:&failure];
if (![read isKindOfClass:NSDictionary.class]) return nil;
NSDictionary *attributes = read[kSnapshotAttributes];
if (![attributes isKindOfClass:NSDictionary.class]) return nil;
id value = attributes[attribute];
return [value isKindOfClass:NSNumber.class] ? @([(NSNumber *)value boolValue]) : nil;
} @catch (NSException *exception) {
(void)exception;
return nil;
}
}

- (nullable NSDictionary *)nodeFromSnapshot:(id)snapshot
namesByNumber:(NSDictionary<NSNumber *, NSString *> *)namesByNumber
interactionOptions:(nullable NSDictionary *)interactionOptions
depth:(NSUInteger)depth
maxDepth:(NSUInteger)maxDepth
maxNodes:(NSUInteger)maxNodes
Expand Down Expand Up @@ -233,6 +263,11 @@ - (nullable NSDictionary *)nodeFromSnapshot:(id)snapshot
id safe = [self jsonValue:attributes[number] name:name];
if (safe) node[name] = safe;
}
if ([node[kAttributeElementType] isEqual:kPresentationDimmingViewClass]) {
NSNumber *interaction = [self userInteractionEnabledForSnapshot:(NSDictionary *)snapshot
options:interactionOptions];
if (interaction) node[kAttributeIsUserInteractionEnabled] = interaction;
}

NSArray *children = ((NSDictionary *)snapshot)[kSnapshotChildren];
if (![children isKindOfClass:NSArray.class]) {
Expand All @@ -246,6 +281,7 @@ - (nullable NSDictionary *)nodeFromSnapshot:(id)snapshot
for (id child in children) {
NSDictionary *built = [self nodeFromSnapshot:child
namesByNumber:namesByNumber
interactionOptions:interactionOptions
depth:depth + 1
maxDepth:maxDepth
maxNodes:maxNodes
Expand Down Expand Up @@ -317,6 +353,15 @@ - (nullable NSDictionary *)snapshotForProcess:(pid_t)pid
options[@"maxDepth"] = @(maxDepth);
options[@"maxChildren"] = @(maxNodes);
options[@"maxArrayCount"] = @(maxNodes);
NSArray<NSNumber *> *interactionAttribute = _attributeNumbersForNames(@[ kAttributeIsUserInteractionEnabled ]);
NSMutableDictionary *interactionOptions = nil;
if ([interactionAttribute isKindOfClass:NSArray.class] && interactionAttribute.count == 1) {
interactionOptions = [options mutableCopy];
interactionOptions[@"attributes"] = interactionAttribute;
interactionOptions[@"maxDepth"] = @0;
interactionOptions[@"maxChildren"] = @0;
interactionOptions[@"maxArrayCount"] = @0;
}
BOOL automationEnabled = [self assertAutomationMode:YES];
NSError *runtimeError = nil;
id snapshot = nil;
Expand Down Expand Up @@ -372,6 +417,7 @@ - (nullable NSDictionary *)snapshotForProcess:(pid_t)pid
NSUInteger count = 0;
NSDictionary *tree = [self nodeFromSnapshot:snapshot
namesByNumber:namesByNumber
interactionOptions:interactionOptions
depth:0
maxDepth:maxDepth
maxNodes:maxNodes
Expand Down
53 changes: 53 additions & 0 deletions docs/adr/0004-ios-snapshot-backend-strategy.md
Original file line number Diff line number Diff line change
Expand Up @@ -475,3 +475,56 @@ last reader that can still refuse geometry it cannot place is the tap-path keybo
width rule therefore remains. The rule detects un-normalized
arrival, not a standing fact about iOS: the producers above do normalize, and a band taller than it
is wide is what one that did not looks like.

## Amendment: modal containment in the presentation cut

The bridge reads one window and reports every container hanging off it, while XCTest's own queries
answer only the presentation the user can reach. React Navigation's card-plus-modal example is the
measured case: three routes presented as sheets left three transition-view containers in one window
and one capture published 567 nodes across three screens, where the same app state under the runner
published 76 across the top screen alone. Maestro's visibility test is geometric, so a `tapOn`
resolved the covered screen's button and the flow failed on an assertion about the screen that never
arrived, and a `presentation: "formSheet"` route's fields matched intermittently (#2638).

Decision. The fold applies modal containment: when the last transition view under a container carries
UIKit's dimming view as its direct child and the producer reports that dimming view takes touches
(`userInteractionEnabled: true`), the earlier transition views whose frame that dimmed area spans are
cut, with their subtrees, from the regular and interactive projections. Every part of the claim is a
producer fact rather than an ordering guess — the dimming view is UIKit's own declaration that it dims
what sits behind, its interaction state says whether a touch there reaches what sits behind, and its
frame is the producer's rectangle, which is what a covered container must be inside. A presentation
with no dimming view, a dimming view the producer did not read or reports passing touches through,
and any container whose earlier sibling is not a transition view keep today's behavior: containment
is asserted only where the producer states it, so the rule fails closed rather than guessing which
siblings are shadows.

A sheet resting at an undimmed detent (`largestUndimmedDetentIdentifier`, react-native-screens
`sheetLargestUndimmedDetentIndex`) is why the interaction state is part of the claim. UIKit keeps the
dimming view in the tree at the same window-sized frame, and the presenting screen stays reachable: on
react-navigation's form-sheet example a press on the presenting screen's `Height Steps` button
navigates while the `Custom Dimming` sheet rests at its smallest detent. The bridge reports that
dimming view `userInteractionEnabled: false`, and `true` for the dimmed form sheet and the card modal.
No attribute the bridge already read told them apart, and `XC_kAXXCAttributeIsVisible` cannot either:
it reads false for the undimmed sheet's dimming view and for the card modal's. The bridge reads the
attribute for dimming views alone, in one follow-up read per view: requesting it on every node cost
about half again the capture time on a 492-node tree, while the targeted read did not move it.
Every source the cut removes is counted in the presentation's
`stats.modalContainedNodeCount` — internal evidence, per ADR 0026, never wire vocabulary — because
the same screen either side of an animation otherwise moves hundreds of comparable lines with nothing
to attribute them to.

Why this seam and not the neighbouring ones. The semantic presentation rules
(`IOS_PRESENTATION_RULES`) run only for the interactive projection, so a suppression rule there could
not deliver the same claim to a regular `snapshot`; the regular eligibility table is pinned against
its Swift twin, so widening it would move a cross-language contract that this rule does not touch; the
occlusion pass *marks* a node covered and keeps it published, and a published-but-marked node is
exactly the 567-versus-76 divergence being fixed, since visibility filtering reads geometry and not
the mark; and cutting at acquisition would leak the decision into `--raw`, which owes the reader the
tree the platform reported. Raw therefore still carries the covered screens.

Producers that report no UIKit class names or no dimming-view interaction state never trigger the
cut, and that is a fact about their output rather than a backend exception in the fold: the runner already omits modal-contained content
from its own queries, `appium-source` and `limrun-ios-tree` report element types and no classes, and
the macOS desktop surface arrives already presented. A scope naming a modal-contained screen now
publishes an empty projection, which is the same healthy empty answer any other unmatched scope gives
— the screen is not what is presented.
2 changes: 2 additions & 0 deletions packages/capture-kit/src/ios-snapshot-engine/engine.ts
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ function presentAcquiredSnapshot(
presentedNodeCount: projected.nodes.length,
sourceNodeCount: acquisition.nodes.length,
parentClipLookups: 0,
modalContainedNodeCount: 0,
},
};
}
Expand Down Expand Up @@ -153,6 +154,7 @@ function presentAcquiredSnapshot(
presentedNodeCount: compacted.nodes.length,
sourceNodeCount: acquisition.nodes.length,
parentClipLookups: folded.stats.parentClipLookups + validation.parentClipLookups,
modalContainedNodeCount: folded.stats.modalContainedNodeCount,
},
};
}
Expand Down
Loading
Loading