Reliability contract: every failed action says whether it ran, scripted taps wait for their target, every platform inherits both

[thymikee]

Why

Test runs on top of agent-device failed in ways nobody could act on. An action timed out, the framework retried it, and the app got two taps. A replayed press fired a render before its button existed and failed, while the same step passed by hand. The error said Element not found or timed out, and the only way to decide "retry or not" was to match message text.

Consumer data (tester-army e2e, mobile.yml, 723 runs, one week): 303 failures from snapshot quality, 111 from session lifecycle, 88 from gestures, 57 from input. The dominant single flake (Login Form, 42 of 68 first-pass failures) is fixed by #3051 and released in v0.21.17. This issue is about the class that remains: actions that run twice, and taps that run too early.

What we get

1. One answer to "is a resend safe?" A failed interaction carries error.details.dispatched: no (nothing reached the device) or unknown (it may have landed). Consumers retry on no, look first on unknown, and stop matching message text. We stop guessing inside the daemon too. There is no yes: no backend can prove execution on its failure path (the runner's own XCTEST_RECORDED_FAILURE says "may not have been performed"), so a third value would only ever be a false claim.
2. No double actions, on any backend. Every internal retry keys on that same proof. Twelve WebDriver routes and the Maestro direct iOS tap resent blindly; they no longer do. The remaining blind resends (daemon no-change retry, runner restart) are tracked below.
3. Scripted taps absorb a render race. A replayed press, click, or longpress waits up to 2 seconds for a target that is not on screen yet, then taps once. Agents and the interactive CLI keep one-shot behavior.
4. A table that fails CI when a backend lies. A row is one way a backend can fail plus the dispatched value that failure must carry (ios-runner.reply.runner-busy → no; a read-only command such as get is always no, because a read has nothing to repeat). For every row, a test makes the real code produce that failure and asserts the value. A row without a test, or a test without a row, fails the gate. When a backend changes what it can prove, the gate says so before a user does.
5. Every platform and provider inherits this. The three behaviors live above the backends: the dispatched seam, the read-only rule, and the readiness wait run in the daemon; the polling engine lives in capture-kit. A new backend gets unknown on every unclassified failure, no on every read, and the readiness wait, with zero backend code. It adds rows only for the refusals it can prove.

Cross-platform: what is inherited, what is declared

flowchart TB
    subgraph consumers [Consumers]
        E2E[e2e frameworks]
        AG[agents via MCP / CLI]
        SDK[Node client]
    end
    subgraph daemon [Daemon: inherited by every backend]
        RD[Readiness wait<br/>replay + Node client, 2 s cap]
        DS[dispatched seam<br/>fills unknown once, never overwrites<br/>9 daemon rows]
        RO[Read-only rule<br/>registry trait → dispatched: no]
        EN[observeUntil engine<br/>capture-kit, one deadline rule]
    end
    subgraph backends [Backends: declare rows, prove them]
        IOS[Apple runner<br/>iOS, tvOS, macOS, visionOS<br/>25 rows]
        AND[Android<br/>adb + helper<br/>9 rows]
        WEB[WebDriver<br/>3 rows]
        NEW[HarmonyOS, Vega, Linux,<br/>next provider<br/>0 rows = all unknown, still safe]
    end
    consumers --> daemon --> backends
    G[(dispatch-disclosure.json<br/>one driver test per producer, gate)]
    G -.enforces.- IOS
    G -.enforces.- AND
    G -.enforces.- WEB

Loading

A backend declares a row when it can prove the action never reached the device (no: a selector miss, a covered target, a runner RUNNER_BUSY, an adb host refusal). Everything else stays unknown, which is safe by construction: a wrong no makes a consumer resend an action that ran; a wrong unknown costs one snapshot. A multi-step action (press --count 25, chunked typing) says no only when its first step was refused; once any step reached the device, a later refusal is unknown.

The readiness wait and the engine need nothing from a backend. They consume the same snapshot a backend already serves.

Performance

• Hit path: one capture, same as today. With the readiness option set, a target present on the first capture is tapped after that capture; measured on a private iPhone 17 Pro simulator, every one of 46 clicks resolved in one poll (waitedMs median 245 ms, which is the capture itself). Per-run wall time was 34.5 s against 36.5 s p50 on a loaded host over 12 pairs, inside the load noise; the soak issue below measures it properly and has a 3% keep rule.
• Miss path: bounded at 2 s, and only where it is set. MCP and the interactive CLI never wait; an agent that misses on a wrong selector gets the fast error it gets today.
• Failure path: cheaper. A WebDriver mutation that timed out used to pay a second full request timeout before failing; it fails after one. dispatched is a field on an error, no round trip.
• Engine: no extra polls. The budget bounds when a poll starts, and a capture in flight is never abandoned mid-way; cancellation joins it. The two loops moved onto the engine (post-gesture stability, scroll movement) keep their cadence and verdicts.
• Request envelope grows only by the budget, on client and daemon, so a replayed step never times out on the client while the daemon is still inside its wait.

Before and after: what a failure tells us

sequenceDiagram
    participant T as Test framework
    participant D as Daemon
    participant B as Backend (runner, adb, WebDriver)
    T->>D: press "Continue"
    D->>B: tap
    B--xD: reply lost
    Note over D: before: error text "timed out"
    D-->>T: error (text only)
    T->>T: regex on message → retry
    T->>D: press "Continue"
    D->>B: tap
    Note over B: the app got two taps

Loading

sequenceDiagram
    participant T as Test framework
    participant D as Daemon
    participant B as Backend (runner, adb, WebDriver)
    T->>D: press "Continue"
    D->>B: tap
    B--xD: reply lost
    Note over D: after: details.dispatched = "unknown"
    D-->>T: error + dispatched
    T->>D: snapshot
    D-->>T: the next screen is up
    Note over T: no retry, the step passed

Loading

error.details.dispatched	Meaning	What a consumer does
"no"	Nothing reached the device	Retry as is
"unknown"	It may have landed	Snapshot before any retry
absent	Not classified	Treat as unknown

Before and after: where retries live

flowchart LR
    subgraph before [Before: five retry sites, five rules]
        W[WebDriver transport<br/>resends 12 mutating routes on timeout]
        M[Maestro direct iOS tap<br/>re-taps on 4 message strings]
        N[Daemon no-change retry<br/>resends when the tree looks the same]
        R[Runner restart<br/>resends after a lost reply]
        A[adb<br/>marks offline retriable, nobody acts]
    end

Loading

flowchart LR
    subgraph after [After: one rule, one table]
        P[Producer classifies its own failure<br/>no / unknown]
        G[(dispatch-disclosure.json<br/>46 rows, one driver test per producer)]
        C{Retry only when<br/>dispatched = no}
        P --> G --> C
        C -->|WebDriver| W2[zero retries on mutations]
        C -->|Maestro direct tap| M2[falls back only on no]
        C -->|adb host refusal| A2[retries once, offline proven pre-dispatch]
        C -->|daemon| N2[no-change retry deleted]
        C -->|runner| R2[restart resend needs proof]
    end

Loading

Before and after: a replayed tap

stateDiagram-v2
    [*] --> Capture
    Capture --> Tap: target present
    Capture --> Fail: target absent
    Tap --> [*]
    Fail --> [*]: selector_not_found
    note right of Fail
        before: one look, one chance
    end note

Loading

stateDiagram-v2
    [*] --> Capture
    Capture --> Tap: target present
    Capture --> FailNow: covered, off-screen, ambiguous
    Capture --> Wait: target absent
    Wait --> Capture: every 200 ms, up to 2 s
    Wait --> Expired: 2 s passed
    Capture --> Sparse: tree went empty
    Tap --> [*]
    FailNow --> [*]: same error as live
    Expired --> [*]: selector_not_found + details.readiness
    Sparse --> [*]: reason capture_sparse

Loading

The wait covers one case only: the target is not there yet. Everything else fails at once, as it does today, on every platform.

Before and after: the polling loops

Three loops (post-gesture stability, scroll movement, readiness) each had their own cadence, deadline, and idea of what "timed out" means. They share one engine, observeUntil: the budget bounds when a poll starts, a capture may overrun by one interval, cancellation joins the running capture, and every poll is on the timeline. Each loop keeps its own verdict (what counts as "done"). A loop a new backend needs (a helper's fill verification, a provider's stability probe) is a verdict plus a schedule, not a new deadline rule.

What does not change

• MCP tools and the interactive CLI: one look, one tap, byte-identical responses.
• No new flag for agents. readinessTimeoutMs is set by replay, or by the Node client.
• dispatched says what happened up to the device. It never claims what the app did next.
• A wait never proves the target stays; a target that vanishes after the poll fails like today.

Delivery

In order:

• fix(provider-webdriver): never resend a mutating WebDriver request #3070 WebDriver: never resend a mutating request (merged)
• feat(errors): disclose whether a failed interaction reached the device #3071 dispatched on failed interactions, table and gate (merged)
• feat(interaction): readiness wait for replay and the Node client on a shared observation engine #3072 Readiness wait for replay and the Node client, on the shared engine (merged)
• refactor(interaction): move post-gesture stability and scroll movement onto observeUntil #3078 Post-gesture stability and scroll movement move onto the engine (merged)
• Delete the no-change retry path (retryOnNoChange, pendingInteractionOutcome) #3073 Delete the daemon no-change retry path → PR fix(daemon): delete the no-change interaction retry path #3083 (on main with feat(errors): disclose whether a failed interaction reached the device #3071)
• iOS runner restart can resend a command that was written and then lost #3074 Runner restart resend needs proof → PR fix(apple-runner): restart without resending a command written then lost #3082 (on main with feat(errors): disclose whether a failed interaction reached the device #3071)
• adb: mark every host refusal as hostRefusal and derive dispatched: "no" from it #3075 adb: hostRefusal on every host refusal, dispatched: no from it → PR fix(android): report dispatched no for adb host refusals #3079 (merged)
• open --launch-url should answer the iOS "Open in <app>?" confirmation #3076 open --launch-url answers the iOS "Open in app?" sheet → PR feat(open): answer the iOS launch confirmation for --launch-url #3084 (merged)
• Readiness wait: nightly soak and keep-or-drop rule #3077 Readiness soak, keep-or-drop rule → code part merged in feat(interaction): report readiness wait on a successful press, click, or longpress #3081; the seven nights start after the next release
• open --launch-url: a post-open target discovery that fails on its deadline must not skip the launch confirmation read #3096 open --launch-url: a post-open target discovery that fails on its deadline must not skip the confirmation read (CI logs of the feat(open): answer the iOS launch confirmation for --launch-url #3084 reds; the first version of the issue blamed the read's budget, which was wrong)
• iOS Simulator open: simctl openurl has no timeout and can hold the open until the daemon request times out #3101 simctl openurl has no timeout (found while reproducing open --launch-url: a post-open target discovery that fails on its deadline must not skip the launch confirmation read #3096)
• refactor(daemon): apply one dispatch-disclosure rule to returned and thrown failures #3099 one disclosure rule in the daemon seam (merged)
• chore: drop unused production exports and suppress dynamic consumers #3100 unused production exports 70 → 50 (merged); the 50 test-only exports stay reported by pnpm check:production-exports
• docs(reliability): state the recovery-tap and WebDriver route rules, drop a deleted retry from a guarantee #3090 docs: recovery-tap rule, WebDriver route rule per request, stale retry clause removed from a guarantee (merged)
• fix(provider-webdriver): an orientation refused by every route reports dispatched no #3091 orientation refused by every WebDriver route reports dispatched: no (merged)

Design record: ADR 0011 gains the targetReadiness and outcomeObservation cells; the golden table lives at contracts/fixtures/dispatch-disclosure.json.

[thymikee]

[claude-fable-5-1] responding on behalf of @thymikee

Where the umbrella stands (2026-10-01)

On main (8c0edab4c7): #3070, #3072, #3081, #3078, #3071 (carrying #3083 and #3082), #3079. Issues #3073, #3074 and #3075 are closed against those commits.

Open: #3084 (open --launch-url answers the iOS sheet; ready for review at d2c7905211, rebased on main, every review thread answered) and #3090 (three documentation statements the arc left unsaid or stale; no behavior change).

What we got, measured

error.details.dispatched	two values, no / unknown; 79-row golden table with a coverage gate; one request ledger that every bound mutation records into; reads always no
Readiness wait	replay and SDK press/click/longpress wait up to 2 s for a late target on the shared observeUntil engine; data.readiness on success, error.details.readiness on a miss
Loops on the engine	post-gesture stability and scroll movement; the WebDriver transport never resends a mutating request; the Apple runner restarts without resending a command it wrote
Package size	+18.7 kB unpacked merged (#3070 1.8, #3072 9.1, #3081 0.5, #3078 0.5, #3071 5.9, #3079 0.9); +5.2 kB more with #3084
Startup	--help medians moved between −5.7 ms and +4.6 ms per PR, inside the run-to-run spread

Live proof ran on AWS Device Farm and BrowserStack (both still serve the legacy activate_app route), private simulators and emulators (dispatched cases, engaged readiness pass and miss, never-settling and slow-capture loops), and a fresh iPhone 17 Pro simulator for the launch sheet (launchConfirmation: "accepted").

What closes the umbrella

1. feat(open): answer the iOS launch confirmation for --launch-url #3084 merges.
2. open --launch-url should answer the iOS "Open in <app>?" confirmation #3076: one unattended run of test/integration/replays/ios/fixture/03-selector-routes-covered-diagnosis.ad on a fresh simulator with a Release fixture that registers agent-device-test-app://. The local fixture build fails on Xcode 26.2 (expo-modules-jsi RuntimeScheduler.h); the CI lane's fixture is the Test App Build Cache artifact fingerprint.<hash>.ios (19.5 MB), so the run needs that artifact or a newer Xcode.
3. Readiness wait: nightly soak and keep-or-drop rule #3077: seven nights of the readiness soak after the next release, then the keep-or-drop decision by its 3 % wall-time rule.
4. docs(reliability): state the recovery-tap and WebDriver route rules, drop a deleted retry from a guarantee #3090 merges.

Later deletions (none block a merge)

• @agent-device/contracts ships a test-only subpath, ./dispatch-disclosure-fixtures, used by four test files; it can move to a test helper.
• discloseDispatchAfterSteps and detailsAfterDispatchedSteps are the thrown and response forms of one rule; they fold once every route fails by throwing.
• The observeUntil knobs minPolls, budgetFrom and captureDeadline each have one caller class today; a knob whose second caller never arrives can become that caller's constant.

Found on the way, outside this umbrella: #3087 (daemon shutdown deletes another daemon's daemon.json).

[thymikee]

[claude-fable-5-1] responding on behalf of @thymikee

Delivery complete (2026-10-01, main c973d3867d)

Every code item is on main: #3070, #3072, #3081, #3078, #3071 (with #3083, #3082), #3079, #3084, #3090, #3091. Issues #3073–#3076 closed. Two items stay open and keep this umbrella open: #3077 (seven-night readiness soak after the next release, then keep-or-drop by the 3 % wall-time rule) and #3096 (the launch-confirmation read should wait for a cold runner under the runner start budget; found by the iOS smoke lane while landing #3084).

What is true on main now

• Every failed interaction says whether it reached the device: error.details.dispatched is no or unknown, from one request ledger that every bound mutation records into, through one router seam, so every platform and provider gets it without its own code. Reads are always no. 80 golden rows (37 no, 43 unknown), each driven by a test, with a gate that fails on an undriven row.
• Three resend paths that could double-act are gone: the daemon no-change tap retry (resolution.ts 1,017 → 53 lines; interaction-outcome-policy.ts, deferred-interaction-outcome.ts deleted), the Apple runner restart resend without proof, the WebDriver mutating resend.
• Replay and SDK press/click/longpress wait up to 2 s for a late target on the shared observeUntil engine; post-gesture stability and scroll movement run on the same engine. data.readiness on a wait that succeeded, error.details.readiness on a miss.
• open --launch-url answers the iOS "Open in app?" sheet for the session app and reports launchConfirmation: "accepted"; the smoke harness no longer answers it.

Cost: production +5,072 / −2,478 lines across 128 files since 0a7221d516; about +24 kB unpacked on a 4.9 MB package; --help and --version medians within the run-to-run spread on every PR.

Proven live: AWS Device Farm and BrowserStack (both still serve the legacy activate_app route), private iOS simulators and Android emulators (dispatched cases, engaged readiness pass and miss, never-settling and slow-capture loops), a fresh iPhone 17 Pro simulator for the sheet (accepted, fixture replay unattended on the CI Release fixture).

Not yet proven: fewer flaky runs in practice. That is #3077's soak and the maintainer's CI on the released version; nothing here measured it.

[thymikee]

[claude-fable-5-1] responding on behalf of @thymikee

Closing: every code item on this list is on main (last two: #3099 75810a1aeb, #3100 9c6ad27678). What stays open has its own issue and says "Part of #3069" so it is still findable from here:

• Readiness wait: nightly soak and keep-or-drop rule #3077 — the seven-night readiness soak after the next release, then keep-or-drop.
• open --launch-url: a post-open target discovery that fails on its deadline must not skip the launch confirmation read #3096 — open --launch-url must read the sheet when the post-open target discovery failed on its deadline (the cause of the feat(open): answer the iOS launch confirmation for --launch-url #3084 CI reds; a confirmable launch that is not observable reads once).
• iOS Simulator open: simctl openurl has no timeout and can hold the open until the daemon request times out #3101 — simctl openurl has no timeout.

Nothing in this arc measured flake rates; #3077 is where that happens.