From b14997ca0982bf071ccf7528c192a4084490e54a Mon Sep 17 00:00:00 2001 From: Thiago Brezinski Date: Fri, 4 Sep 2026 00:08:56 +0100 Subject: [PATCH 1/8] test(ci): route iOS 26 device-contract pointer scenarios through Detox The first pointer gesture the Agent Device XCTest runner synthesizes into a freshly erased, cold-booted iOS 26.5 simulator on hosted runners arrives seconds late or never while injection reports success (#54), which is the release pipeline's dominant failure. The ios26.auto-fallback cell's unique responsibility is proving automatic engine selection picks the fallback engine, and Detox pointer synthesis proves that equally. Delete the five ios26.auto-fallback pointer overrides so they fall through to the default Detox driver, and flip the matching route-table entries, which is what scripts/run-device-contract-isolated.mjs dispatches on. Scenario definitions, terminal public-outcome assertions, and the four-engine parity comparison are unchanged. Add a test asserting every route equals the driver its pointer plan resolves to, so the route table and the override map cannot drift into starting a harness that has no plan for the scenario. Refs #65 Co-Authored-By: Claude Opus 5 (1M context) --- .../0006-automated-publication-confidence.md | 11 ++++ e2e/contracts/pointer-drivers.json | 52 ++----------------- src/__tests__/issue39-device-contract.test.ts | 45 ++++++++++++---- 3 files changed, 51 insertions(+), 57 deletions(-) diff --git a/docs/adr/0006-automated-publication-confidence.md b/docs/adr/0006-automated-publication-confidence.md index 5f6164a..8be43b7 100644 --- a/docs/adr/0006-automated-publication-confidence.md +++ b/docs/adr/0006-automated-publication-confidence.md @@ -22,3 +22,14 @@ Amended 2026-09-02 ([#57](https://github.com/thiagobrez/react-native-reorderable | Publication | `release.yml`, triggered by `workflow_run` on candidate success | Release | Pruned as redundant or unused: the candidate's duplicate `quality` and `render-regressions` jobs, `ci.yml`'s `build-library` job (a strict subset of `test`), all `merge_group` triggers (no merge queue is configured), coverage collection (collected twice, enforced nowhere), the duplicate package-artifact verification and sha256 re-checks, and the manual named-scenario device-verification scripts. Physical-device performance validation is removed from the lifecycle entirely: Reassure render gates plus geometry benchmarks on hosted CI are the accepted performance proof, and device-specific performance regressions join the dogfooding-caught risk class ([#45](https://github.com/thiagobrez/react-native-reorderable/issues/45) closed accordingly). The required checks on `main` become `lint`, `test`, `build-android`, `build-ios`, the documentation `build`, `geometry`, `render-regressions`, and the changeset requirement; `build-library` and the tarball-approval check leave the PR gate. + +Amended 2026-09-04 ([#65](https://github.com/thiagobrez/react-native-reorderable/issues/65), decided in [#60](https://github.com/thiagobrez/react-native-reorderable/issues/60)): pointer harness ownership is now fixed per device-contract configuration and scenario rather than preferring Agent Device everywhere it can express the gesture. + +| Configuration | Pointer harness | +| --- | --- | +| `ios27.native` | Agent Device, except `multi-selection-reorder` on Detox | +| `ios27.fallback` | Agent Device | +| `ios26.auto-fallback` | Detox | +| `android.fallback` | Detox | + +The iOS 26 cell moves to Detox because the first pointer gesture the Agent Device XCTest runner synthesizes into a freshly erased, cold-booted iOS 26.5 simulator on hosted runners arrives seconds late or never while injection reports success ([#54](https://github.com/thiagobrez/react-native-reorderable/issues/54)) — the pipeline's dominant failure, and one no widened gesture timing or scenario retry may paper over, because a silent input drop is indistinguishable from a broken contract. That cell's unique responsibility is proving automatic engine selection picks the fallback engine on a runtime without native SwiftUI reordering, and Detox pointer synthesis proves that equally: scenario definitions, terminal public-outcome assertions, and the four-engine parity comparison are unchanged. Agent Device is retained on iOS 27, where its selector-targeted center drag is the only harness that expresses insertion against native SwiftUI rows. diff --git a/e2e/contracts/pointer-drivers.json b/e2e/contracts/pointer-drivers.json index 889ff1d..59d69ce 100644 --- a/e2e/contracts/pointer-drivers.json +++ b/e2e/contracts/pointer-drivers.json @@ -97,48 +97,6 @@ } } }, - "ios26.auto-fallback": { - "free-form-reorder": { - "driver": "agent-device", - "semanticTargetLabel": "Card row 4", - "destinationSelector": { - "relation": "namedTargetCenterAfterSourceLift", - "label": "Card row 4" - } - }, - "virtualized-list-reorder": { - "driver": "agent-device", - "semanticTargetLabel": "List row 4", - "destinationSelector": { - "relation": "namedTargetCenterAfterSourceLift", - "label": "List row 4" - } - }, - "section-list-reorder": { - "driver": "agent-device", - "semanticTargetLabel": "Section 1, row 4", - "destinationSelector": { - "relation": "namedTargetCenterAfterSourceLift", - "label": "Section 1, row 4" - } - }, - "multi-selection-reorder": { - "driver": "agent-device", - "semanticTargetLabel": "Card row 6", - "destinationSelector": { - "relation": "namedTargetCenterAfterSourceLift", - "label": "Card row 6" - } - }, - "scoped-drop": { - "driver": "agent-device", - "semanticTargetLabel": "Accepting drop zone", - "destinationSelector": { - "relation": "namedTargetCenter", - "label": "Accepting drop zone" - } - } - }, "android.fallback": { "free-form-reorder": { "driver": "detox", @@ -198,11 +156,11 @@ "scoped-drop": "agent-device" }, "ios26.auto-fallback": { - "free-form-reorder": "agent-device", - "virtualized-list-reorder": "agent-device", - "section-list-reorder": "agent-device", - "multi-selection-reorder": "agent-device", - "scoped-drop": "agent-device" + "free-form-reorder": "detox", + "virtualized-list-reorder": "detox", + "section-list-reorder": "detox", + "multi-selection-reorder": "detox", + "scoped-drop": "detox" }, "android.fallback": { "free-form-reorder": "detox", diff --git a/src/__tests__/issue39-device-contract.test.ts b/src/__tests__/issue39-device-contract.test.ts index f06431a..903814d 100644 --- a/src/__tests__/issue39-device-contract.test.ts +++ b/src/__tests__/issue39-device-contract.test.ts @@ -114,17 +114,14 @@ describe('issue 39 portable device contract', () => { destination: { normalizedX: 0.5, normalizedY: 0.25 }, driver: 'detox', }); - expect(pointerDrivers.routes['ios26.auto-fallback']).toMatchObject({ - 'free-form-reorder': 'agent-device', - 'multi-selection-reorder': 'agent-device', - 'scoped-drop': 'agent-device', - 'section-list-reorder': 'agent-device', - 'virtualized-list-reorder': 'agent-device', - }); - expect(pointerDrivers.overrides['ios26.auto-fallback']).toMatchObject({ - 'free-form-reorder': { driver: 'agent-device' }, - 'multi-selection-reorder': { driver: 'agent-device' }, + expect(pointerDrivers.routes['ios26.auto-fallback']).toEqual({ + 'free-form-reorder': 'detox', + 'virtualized-list-reorder': 'detox', + 'section-list-reorder': 'detox', + 'multi-selection-reorder': 'detox', + 'scoped-drop': 'detox', }); + expect(pointerDrivers.overrides['ios26.auto-fallback']).toBeUndefined(); expect(pointerDrivers.routes['android.fallback']).toEqual({ 'free-form-reorder': 'detox', 'virtualized-list-reorder': 'detox', @@ -170,6 +167,34 @@ describe('issue 39 portable device contract', () => { expect(runner).not.toMatch(/\.skip\(|\.todo\(|retryTimes|--retries/); }); + it('routes every configuration to the harness its pointer plan resolves to', () => { + const pointerDrivers = JSON.parse( + read('e2e/contracts/pointer-drivers.json') + ) as { + default: { driver: string }; + overrides: Record>; + routes: Record>; + }; + // The isolated runner dispatches on `routes`, while the Agent Device + // pointer runner reads `overrides` and refuses a scenario it has no plan + // for. A configuration whose route disagrees with its resolved plan starts + // a harness that cannot run the scenario. + const resolved = Object.fromEntries( + Object.entries(pointerDrivers.routes).map(([configuration, routes]) => [ + configuration, + Object.fromEntries( + Object.keys(routes).map((scenarioId) => [ + scenarioId, + pointerDrivers.overrides[configuration]?.[scenarioId]?.driver ?? + pointerDrivers.default.driver, + ]) + ), + ]) + ); + + expect(resolved).toEqual(pointerDrivers.routes); + }); + it('plans native SwiftUI insertion by semantic target when the driver is center-only', () => { const scenarios = JSON.parse( read('e2e/contracts/scenarios.json') From c6331e5d79c492dad8bf87b06907aa7396a5a7b0 Mon Sep 17 00:00:00 2001 From: Thiago Brezinski Date: Fri, 4 Sep 2026 10:41:42 +0100 Subject: [PATCH 2/8] ci: excise the screenshot-feedback apparatus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pulled forward from the CI overhaul plan (4/9, #68) because the first Detox-routed iOS 26 contract run failed inside this apparatus, and the tarball-approval required check gates on the contract, feedback, and parity jobs — nothing can merge to main until the iOS 26 job is green, including the PRs that were scheduled to delete this code. The failing assertion ("drag released before feedback sampling") did not mean the drag was short: the five drag scenarios held the destination for 8000ms, well past the ~4.9s sampling window. It fired because the Detox gesture promise settled early and the sampling assert masked the real error, which the catch had swallowed. Remove the proof end to end, as decided in #55 and recorded in ADR 0006 (2026-09-01 amendment): the feedback workflow job and its approve-candidate edge, the Vision OCR analyzer, the report verifier and its specs and fixtures, the mid-hold sampling in both pointer runners, the mtime-poll IPC and its baseline/gesture-marker/terminal screenshots in the replay scripts, the sampling constants in pointer-timing.json, and the feedback evidence category in the retry shells. Drag scenarios drop their 8000ms destination hold and use the gesture's own 2000ms default; the Agent Device replay gesture timing is unchanged. The deep-link confirmation marker, previously written when the baseline screenshot appeared, is written on successful replay exit instead. Detox per-test video and screenshot recording stays as the human-reviewable visual record. The exit-75 infra/contract retry protocol, per-attempt evidence isolation, and the four-engine parity comparison are untouched. Refs #68 #65 Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/exact-package-candidate.yml | 42 --- e2e/contracts/feedback-specs.json | 47 --- .../fixtures/feedback-ambiguous.json | 41 --- .../fixtures/feedback-baseline-ambiguous.json | 18 -- .../feedback-insufficient-change.json | 21 -- e2e/contracts/fixtures/feedback-missing.json | 13 - .../fixtures/feedback-occluded-valid.json | 21 -- .../fixtures/feedback-source-outside.json | 54 ---- e2e/contracts/fixtures/feedback-valid.json | 108 ------- e2e/contracts/outcome-observation.cjs | 7 - e2e/contracts/pointer-timing.json | 7 +- e2e/contracts/portable-contract.e2e.cjs | 52 +-- e2e/contracts/scenarios.json | 15 +- scripts/analyze-device-feedback.swift | 297 ------------------ scripts/run-agent-device-pointer.mjs | 84 +---- scripts/run-device-contract-isolated.mjs | 7 - scripts/run-device-contract-job.mjs | 16 +- scripts/verify-device-feedback.mjs | 182 ----------- src/__tests__/issue39-device-contract.test.ts | 277 +--------------- 19 files changed, 29 insertions(+), 1280 deletions(-) delete mode 100644 e2e/contracts/feedback-specs.json delete mode 100644 e2e/contracts/fixtures/feedback-ambiguous.json delete mode 100644 e2e/contracts/fixtures/feedback-baseline-ambiguous.json delete mode 100644 e2e/contracts/fixtures/feedback-insufficient-change.json delete mode 100644 e2e/contracts/fixtures/feedback-missing.json delete mode 100644 e2e/contracts/fixtures/feedback-occluded-valid.json delete mode 100644 e2e/contracts/fixtures/feedback-source-outside.json delete mode 100644 e2e/contracts/fixtures/feedback-valid.json delete mode 100644 scripts/analyze-device-feedback.swift delete mode 100644 scripts/verify-device-feedback.mjs diff --git a/.github/workflows/exact-package-candidate.yml b/.github/workflows/exact-package-candidate.yml index 67360ce..48daed7 100644 --- a/.github/workflows/exact-package-candidate.yml +++ b/.github/workflows/exact-package-candidate.yml @@ -619,48 +619,8 @@ jobs: artifacts/issue-39/device-tables/${{ matrix.configuration }}*.json artifacts/issue-39/outcomes-by-case/${{ matrix.configuration }}/ artifacts/issue-39/outcomes/${{ matrix.configuration }}.json - artifacts/issue-39/feedback/${{ matrix.configuration }}/ artifacts/issue-39/agent-device/${{ matrix.configuration }}/ - feedback: - name: Verify continuous public destination feedback - if: always() - needs: contract - runs-on: macos-26 - steps: - - name: Checkout - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 - - - name: Download four-engine feedback artifacts - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 - with: - pattern: issue-39-* - path: artifacts/issue-39/downloads - merge-multiple: true - - - name: Fail if any producer failed - if: needs.contract.result != 'success' - run: exit 1 - - - name: Extract public labels and destination visual change - run: | - swift scripts/analyze-device-feedback.swift \ - artifacts/issue-39/downloads/feedback \ - e2e/contracts/feedback-specs.json \ - artifacts/issue-39/feedback-report.json - - - name: Verify every held-pointer sample - run: node scripts/verify-device-feedback.mjs artifacts/issue-39/feedback-report.json - - - name: Upload semantic feedback report - if: always() - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 - with: - name: issue-39-feedback-report - if-no-files-found: error - retention-days: 14 - path: artifacts/issue-39/feedback-report.json - parity: name: Compare four-engine public outcomes if: always() @@ -694,7 +654,6 @@ jobs: needs.render-regressions.result == 'success' && needs.clean-native-consumers.result == 'success' && needs.contract.result == 'success' && - needs.feedback.result == 'success' && needs.parity.result == 'success' && ( (github.event_name != 'schedule' && github.event_name != 'workflow_dispatch' && @@ -712,7 +671,6 @@ jobs: - minimum-android-runtime - minimum-ios-runtime - contract - - feedback - parity runs-on: ubuntu-latest steps: diff --git a/e2e/contracts/feedback-specs.json b/e2e/contracts/feedback-specs.json deleted file mode 100644 index 3b81879..0000000 --- a/e2e/contracts/feedback-specs.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "configurations": [ - "ios27.native", - "ios27.fallback", - "ios26.auto-fallback", - "android.fallback" - ], - "scenarios": { - "free-form-reorder": { - "kind": "reorder", - "sourceLabel": "Card row 1", - "predecessorLabel": "Card row 3", - "targetLabel": "Card row 4" - }, - "virtualized-list-reorder": { - "kind": "reorder", - "sourceLabel": "List row 1", - "sourceLabelAliasesByConfiguration": { - "android.fallback": ["List row"] - }, - "predecessorLabel": "List row 3", - "targetLabel": "List row 4" - }, - "section-list-reorder": { - "kind": "reorder", - "sourceLabel": "Section 1, row 1", - "predecessorLabel": "Section 1, row 3", - "targetLabel": "Section 1, row 4" - }, - "multi-selection-reorder": { - "kind": "reorder", - "sourceLabel": "Card row 2", - "predecessorLabel": "Card row 4", - "targetLabel": "Card row 6" - }, - "scoped-drop": { - "kind": "drop", - "targetLabel": "Accepting drop zone", - "visualTargetLabel": "Acceptin", - "visualTargetLabelsByConfiguration": { - "ios27.fallback": "Drop selected items here", - "ios26.auto-fallback": "Drop selected items here", - "android.fallback": "Drop selected items here" - } - } - } -} diff --git a/e2e/contracts/fixtures/feedback-ambiguous.json b/e2e/contracts/fixtures/feedback-ambiguous.json deleted file mode 100644 index 4815d38..0000000 --- a/e2e/contracts/fixtures/feedback-ambiguous.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "runs": [ - { - "configuration": "ios27.native", - "scenario": "free-form-reorder", - "kind": "reorder", - "sourceLabel": "Card row 1", - "predecessorLabel": "Card row 3", - "targetLabel": "Card row 4", - "baselineLabels": [ - { "text": "Card row 3", "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } }, - { "text": "Card row 4", "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } } - ], - "samples": [ - { - "path": "sample-1.png", - "labels": [ - { - "text": "Card row 3", - "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } - }, - { - "text": "Card row 1", - "bounds": { "x": 0, "y": 0.4, "width": 1, "height": 0.1 } - }, - { - "text": "Card row 1", - "bounds": { "x": 0, "y": 0.45, "width": 1, "height": 0.1 } - }, - { - "text": "Card row 4", - "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } - } - ] - }, - { "path": "sample-2.png", "labels": [] }, - { "path": "sample-3.png", "labels": [] } - ] - } - ] -} diff --git a/e2e/contracts/fixtures/feedback-baseline-ambiguous.json b/e2e/contracts/fixtures/feedback-baseline-ambiguous.json deleted file mode 100644 index 4c529f6..0000000 --- a/e2e/contracts/fixtures/feedback-baseline-ambiguous.json +++ /dev/null @@ -1,18 +0,0 @@ -{ - "runs": [ - { - "configuration": "ios27.native", - "scenario": "multi-selection-reorder", - "kind": "reorder", - "sourceLabel": "Card row 2", - "predecessorLabel": "Card row 4", - "targetLabel": "Card row 6", - "baselineLabels": [ - { "text": "Card row 4", "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } }, - { "text": "Card row 6", "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } }, - { "text": "Card row 6", "bounds": { "x": 0, "y": 0.7, "width": 1, "height": 0.1 } } - ], - "samples": [{ "path": "sample-1.png", "labels": [] }, { "path": "sample-2.png", "labels": [] }, { "path": "sample-3.png", "labels": [] }] - } - ] -} diff --git a/e2e/contracts/fixtures/feedback-insufficient-change.json b/e2e/contracts/fixtures/feedback-insufficient-change.json deleted file mode 100644 index 2a135ad..0000000 --- a/e2e/contracts/fixtures/feedback-insufficient-change.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "runs": [ - { - "configuration": "ios27.native", - "scenario": "multi-selection-reorder", - "kind": "reorder", - "sourceLabel": "Card row 2", - "predecessorLabel": "Card row 4", - "targetLabel": "Card row 6", - "baselineLabels": [ - { "text": "Card row 4", "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } }, - { "text": "Card row 6", "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } } - ], - "samples": [ - { "path": "sample-1.png", "destinationChangeRatio": 0.01, "labels": [{ "text": "Card row 2", "bounds": { "x": 0.2, "y": 0.43, "width": 0.6, "height": 0.05 } }] }, - { "path": "sample-2.png", "destinationChangeRatio": 0.08, "labels": [{ "text": "Card row 2", "bounds": { "x": 0.2, "y": 0.44, "width": 0.6, "height": 0.05 } }] }, - { "path": "sample-3.png", "destinationChangeRatio": 0.08, "labels": [{ "text": "Card row 2", "bounds": { "x": 0.2, "y": 0.45, "width": 0.6, "height": 0.05 } }] } - ] - } - ] -} diff --git a/e2e/contracts/fixtures/feedback-missing.json b/e2e/contracts/fixtures/feedback-missing.json deleted file mode 100644 index daede72..0000000 --- a/e2e/contracts/fixtures/feedback-missing.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "runs": [ - { - "configuration": "ios27.native", - "scenario": "free-form-reorder", - "kind": "reorder", - "sourceLabel": "Card row 1", - "predecessorLabel": "Card row 3", - "targetLabel": "Card row 4", - "samples": [] - } - ] -} diff --git a/e2e/contracts/fixtures/feedback-occluded-valid.json b/e2e/contracts/fixtures/feedback-occluded-valid.json deleted file mode 100644 index 958e15f..0000000 --- a/e2e/contracts/fixtures/feedback-occluded-valid.json +++ /dev/null @@ -1,21 +0,0 @@ -{ - "runs": [ - { - "configuration": "ios27.native", - "scenario": "multi-selection-reorder", - "kind": "reorder", - "sourceLabel": "Card row 2", - "predecessorLabel": "Card row 4", - "targetLabel": "Card row 6", - "baselineLabels": [ - { "text": "Card row 4", "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } }, - { "text": "Card row 6", "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } } - ], - "samples": [ - { "path": "sample-1.png", "destinationChangeRatio": 0.08, "labels": [{ "text": "Card row 2", "bounds": { "x": 0.2, "y": 0.43, "width": 0.6, "height": 0.05 } }] }, - { "path": "sample-2.png", "destinationChangeRatio": 0.07, "labels": [{ "text": "Card row 2", "bounds": { "x": 0.2, "y": 0.44, "width": 0.6, "height": 0.05 } }] }, - { "path": "sample-3.png", "destinationChangeRatio": 0.09, "labels": [{ "text": "Card row 2", "bounds": { "x": 0.2, "y": 0.45, "width": 0.6, "height": 0.05 } }] } - ] - } - ] -} diff --git a/e2e/contracts/fixtures/feedback-source-outside.json b/e2e/contracts/fixtures/feedback-source-outside.json deleted file mode 100644 index 49506e1..0000000 --- a/e2e/contracts/fixtures/feedback-source-outside.json +++ /dev/null @@ -1,54 +0,0 @@ -{ - "runs": [ - { - "configuration": "ios27.native", - "scenario": "multi-selection-reorder", - "kind": "reorder", - "sourceLabel": "Card row 2", - "predecessorLabel": "Card row 4", - "targetLabel": "Card row 6", - "baselineLabels": [ - { - "text": "Card row 4", - "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } - }, - { - "text": "Card row 6", - "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } - } - ], - "samples": [ - { - "path": "sample-1.png", - "destinationChangeRatio": 0.08, - "labels": [ - { - "text": "Card row 2", - "bounds": { "x": 0.2, "y": 0.72, "width": 0.6, "height": 0.05 } - } - ] - }, - { - "path": "sample-2.png", - "destinationChangeRatio": 0.08, - "labels": [ - { - "text": "Card row 2", - "bounds": { "x": 0.2, "y": 0.72, "width": 0.6, "height": 0.05 } - } - ] - }, - { - "path": "sample-3.png", - "destinationChangeRatio": 0.08, - "labels": [ - { - "text": "Card row 2", - "bounds": { "x": 0.2, "y": 0.45, "width": 0.6, "height": 0.05 } - } - ] - } - ] - } - ] -} diff --git a/e2e/contracts/fixtures/feedback-valid.json b/e2e/contracts/fixtures/feedback-valid.json deleted file mode 100644 index b913155..0000000 --- a/e2e/contracts/fixtures/feedback-valid.json +++ /dev/null @@ -1,108 +0,0 @@ -{ - "runs": [ - { - "configuration": "ios27.native", - "scenario": "free-form-reorder", - "kind": "reorder", - "sourceLabel": "Card row 1", - "predecessorLabel": "Card row 3", - "targetLabel": "Card row 4", - "baselineLabels": [ - { "text": "Card row 3", "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } }, - { "text": "Card row 4", "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } } - ], - "samples": [ - { - "path": "sample-1.png", - "labels": [ - { - "text": "Card row 3", - "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } - }, - { - "text": "Card row 1", - "bounds": { "x": 0.2, "y": 0.45, "width": 0.6, "height": 0.05 } - }, - { - "text": "Card row 4", - "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } - } - ] - }, - { - "path": "sample-2.png", - "labels": [ - { - "text": "Card row 3", - "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } - }, - { - "text": "Card row 1", - "bounds": { "x": 0.2, "y": 0.46, "width": 0.6, "height": 0.05 } - }, - { - "text": "Card row 4", - "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } - } - ] - }, - { - "path": "sample-3.png", - "labels": [ - { - "text": "Card row 3", - "bounds": { "x": 0, "y": 0.3, "width": 1, "height": 0.1 } - }, - { - "text": "Card row 1", - "bounds": { "x": 0.2, "y": 0.44, "width": 0.6, "height": 0.05 } - }, - { - "text": "Card row 4", - "bounds": { "x": 0, "y": 0.6, "width": 1, "height": 0.1 } - } - ] - } - ] - }, - { - "configuration": "android.fallback", - "scenario": "scoped-drop", - "kind": "drop", - "targetLabel": "Accepting drop zone", - "baselineLabels": [], - "samples": [ - { - "path": "drop-1.png", - "destinationChangeRatio": 0.08, - "labels": [ - { - "text": "Accepting drop zone", - "bounds": { "x": 0, "y": 0.5, "width": 1, "height": 0.2 } - } - ] - }, - { - "path": "drop-2.png", - "destinationChangeRatio": 0.07, - "labels": [ - { - "text": "Accepting drop zone", - "bounds": { "x": 0, "y": 0.5, "width": 1, "height": 0.2 } - } - ] - }, - { - "path": "drop-3.png", - "destinationChangeRatio": 0.09, - "labels": [ - { - "text": "Accepting drop zone", - "bounds": { "x": 0, "y": 0.5, "width": 1, "height": 0.2 } - } - ] - } - ] - } - ] -} diff --git a/e2e/contracts/outcome-observation.cjs b/e2e/contracts/outcome-observation.cjs index 6181231..46ea629 100644 --- a/e2e/contracts/outcome-observation.cjs +++ b/e2e/contracts/outcome-observation.cjs @@ -26,16 +26,13 @@ function observedLabelsFromSnapshot(snapshot, expectedLabels, bundleId) { function agentDeviceReplayScript({ acceptDeepLinkPrompt, - baselinePath, deepLink, destinationSelector, expectedLabels, - gestureMarkerPath, initialLabels, platform, recordingPath, sourceSelector, - terminalPath, timing, }) { const quote = (value) => `"${value.replaceAll('"', '\\"')}"`; @@ -51,15 +48,11 @@ function agentDeviceReplayScript({ 'open "${DEEP_LINK}"', ...(acceptDeepLinkPrompt ? ['alert accept', 'open "${DEEP_LINK}"'] : []), ...initialLabels.map((label) => `wait ${quote(label)} 15000`), - `screenshot ${quote(baselinePath)}`, ...(recordingPath == null ? [] : [`record start ${quote(recordingPath)} --scope device`]), - 'wait 1000', - `screenshot ${quote(gestureMarkerPath)}`, `gesture drag ${quote(sourceSelector)} ${quote(destinationSelector)} ${timing.sourceHoldMs} ${timing.moveBudgetMs} ${timing.destinationHoldMs}`, ...expectedLabels.map((label) => `wait ${quote(label)} 15000`), - `screenshot ${quote(terminalPath)}`, ...(recordingPath == null ? [] : ['record stop']), '', ].join('\n'); diff --git a/e2e/contracts/pointer-timing.json b/e2e/contracts/pointer-timing.json index a98513d..f218a7f 100644 --- a/e2e/contracts/pointer-timing.json +++ b/e2e/contracts/pointer-timing.json @@ -1,10 +1,5 @@ { "sourceHoldMs": 650, "moveBudgetMs": 1200, - "settleMarginMs": 1650, - "agentDeviceSettleMarginMs": 3500, - "predecessorCenterSettleMarginMs": 5000, - "destinationHoldMs": 8000, - "sampleCount": 3, - "sampleIntervalMs": 700 + "destinationHoldMs": 8000 } diff --git a/e2e/contracts/portable-contract.e2e.cjs b/e2e/contracts/portable-contract.e2e.cjs index f4744cd..f31e4a8 100644 --- a/e2e/contracts/portable-contract.e2e.cjs +++ b/e2e/contracts/portable-contract.e2e.cjs @@ -1,4 +1,3 @@ -const assert = require('node:assert/strict'); const { execFile } = require('node:child_process'); const { mkdir, writeFile } = require('node:fs/promises'); const { dirname } = require('node:path'); @@ -29,10 +28,6 @@ const visible = (label) => element(by.label(label)); // A selector resolves to its center, but the portable action is specifically // insertion before the named target. The upper quarter stays on the target // while selecting its leading insertion side in every engine. -const feedbackFirstSampleMs = - pointerTiming.sourceHoldMs + - pointerTiming.moveBudgetMs + - pointerTiming.settleMarginMs; async function scenarioElement(scenario, label) { const publicElement = visible(label); @@ -68,34 +63,9 @@ async function openScenario(scenario) { return { labels: initialLabels }; } -async function captureOsScreenshot(scenarioId, name) { - const directory = - process.env.ISSUE39_FEEDBACK_DIR ?? - `artifacts/issue-39/feedback/${configuration}/${scenarioId}`; - const path = `${directory}/${name}.png`; - await mkdir(directory, { recursive: true }); - if (platform === 'ios') { - await execFileAsync('xcrun', [ - 'simctl', - 'io', - device.id, - 'screenshot', - path, - ]); - } else { - const { stdout } = await execFileAsync( - 'adb', - ['-s', device.id, 'exec-out', 'screencap', '-p'], - { encoding: 'buffer', maxBuffer: 16 * 1024 * 1024 } - ); - await writeFile(path, stdout); - } - return path; -} - async function runUninterruptedDrag( scenario, - observeFeedback, + throwOnGestureRejection, onActionStarted ) { if ( @@ -111,10 +81,6 @@ async function runUninterruptedDrag( const destination = pointerDrivers.default.destination; await detoxExpectVisible(source); await detoxExpectVisible(target); - if (observeFeedback) { - await captureOsScreenshot(scenario.id, 'baseline'); - } - let completed = false; let rejected; const action = source .longPressAndDrag( @@ -131,25 +97,11 @@ async function runUninterruptedDrag( ) .catch((error) => { rejected = error; - }) - .finally(() => { - completed = true; }); onActionStarted?.(); - if (observeFeedback) { - await delay(feedbackFirstSampleMs); - for (let sample = 0; sample < pointerTiming.sampleCount; sample += 1) { - assert.equal(completed, false, 'drag released before feedback sampling'); - await captureOsScreenshot(scenario.id, `sample-${sample + 1}`); - if (sample + 1 < pointerTiming.sampleCount) { - await delay(pointerTiming.sampleIntervalMs); - } - } - } - await action; - if (observeFeedback && rejected) throw rejected; + if (throwOnGestureRejection && rejected) throw rejected; return rejected; } diff --git a/e2e/contracts/scenarios.json b/e2e/contracts/scenarios.json index aa3608f..94bf069 100644 --- a/e2e/contracts/scenarios.json +++ b/e2e/contracts/scenarios.json @@ -7,8 +7,7 @@ "kind": "drag", "destinationPlacement": "beforeTarget", "sourceLabel": "Card row 1", - "targetLabel": "Card row 4", - "holdDurationMs": 8000 + "targetLabel": "Card row 4" }, "covers": ["free-form"], "selectors": [ @@ -43,8 +42,7 @@ "kind": "drag", "destinationPlacement": "beforeTarget", "sourceLabel": "List row 1", - "targetLabel": "List row 4", - "holdDurationMs": 8000 + "targetLabel": "List row 4" }, "covers": ["list"], "selectors": [ @@ -71,8 +69,7 @@ "kind": "drag", "destinationPlacement": "beforeTarget", "sourceLabel": "Section 1, row 1", - "targetLabel": "Section 1, row 4", - "holdDurationMs": 8000 + "targetLabel": "Section 1, row 4" }, "covers": ["section-list"], "selectors": [ @@ -99,8 +96,7 @@ "kind": "drag", "destinationPlacement": "beforeTarget", "sourceLabel": "Card row 2", - "targetLabel": "Card row 6", - "holdDurationMs": 8000 + "targetLabel": "Card row 6" }, "covers": ["multi-selection"], "selectors": [ @@ -127,8 +123,7 @@ "kind": "drag", "destinationPlacement": "beforeTarget", "sourceLabel": "blue draggable card, selected", - "targetLabel": "Accepting drop zone", - "holdDurationMs": 8000 + "targetLabel": "Accepting drop zone" }, "covers": ["scoped-drag-drop"], "selectors": [ diff --git a/scripts/analyze-device-feedback.swift b/scripts/analyze-device-feedback.swift deleted file mode 100644 index 2ddd1d6..0000000 --- a/scripts/analyze-device-feedback.swift +++ /dev/null @@ -1,297 +0,0 @@ -import AppKit -import Foundation -import Vision - -struct Bounds: Codable { - let x: Double - let y: Double - let width: Double - let height: Double -} - -struct Label: Codable { - let text: String - let bounds: Bounds -} - -struct Sample: Codable { - let path: String - let labels: [Label] - let destinationChangeRatio: Double? -} - -struct Run: Codable { - let configuration: String - let scenario: String - let kind: String - let sourceLabel: String? - let sourceLabelAliases: [String]? - let predecessorLabel: String? - let targetLabel: String - let visualTargetLabel: String? - let baselineLabels: [Label] - let samples: [Sample] -} - -struct Report: Codable { let runs: [Run] } -struct ScenarioSpec: Decodable { - let kind: String - let sourceLabel: String? - let sourceLabelAliasesByConfiguration: [String: [String]]? - let predecessorLabel: String? - let targetLabel: String - let visualTargetLabel: String? - let visualTargetLabelsByConfiguration: [String: String]? -} -struct Specs: Decodable { - let configurations: [String] - let scenarios: [String: ScenarioSpec] -} - -enum FeedbackError: Error, CustomStringConvertible { - case message(String) - var description: String { - switch self { case let .message(value): value } - } -} - -func recognize(_ url: URL) throws -> [Label] { - guard let image = NSImage(contentsOf: url), - let cgImage = image.cgImage(forProposedRect: nil, context: nil, hints: nil) - else { throw FeedbackError.message("Cannot load \(url.path)") } - let request = VNRecognizeTextRequest() - request.recognitionLevel = .accurate - request.usesLanguageCorrection = false - try VNImageRequestHandler(cgImage: cgImage).perform([request]) - return (request.results ?? []).compactMap { observation in - guard let candidate = observation.topCandidates(1).first else { return nil } - let box = observation.boundingBox - return Label( - text: candidate.string, - bounds: Bounds( - x: box.origin.x, y: box.origin.y, - width: box.width, height: box.height - ) - ) - } -} - -func exactLabel(_ labels: [Label], _ text: String, path: String) throws -> Label { - func comparable(_ value: String) -> String { - let trimmed = value.trimmingCharacters(in: .whitespacesAndNewlines) - // Vision can read the adjacent dashed destination border as a leading - // apostrophe at simulator point resolution. Remove only that observed - // prefix; the public label itself remains an exact match. - return trimmed.hasPrefix("' ") ? String(trimmed.dropFirst(2)) : trimmed - } - let matches = labels.filter { - comparable($0.text).localizedCaseInsensitiveCompare(comparable(text)) - == .orderedSame - } - guard matches.count == 1, let match = matches.first else { - throw FeedbackError.message( - "\(path): expected one OCR label \(text.debugDescription), found \(matches.count)" - ) - } - return match -} - -func decodedImage(_ url: URL) throws -> CGImage { - guard let image = NSImage(contentsOf: url), - let cg = image.cgImage(forProposedRect: nil, context: nil, hints: nil) - else { throw FeedbackError.message("Cannot load pixels from \(url.path)") } - return cg -} - -func rgba( - _ image: CGImage, width: Int? = nil, height: Int? = nil -) throws -> (bytes: [UInt8], width: Int, height: Int) { - let width = width ?? image.width - let height = height ?? image.height - let stride = width * 4 - var bytes = [UInt8](repeating: 0, count: stride * height) - guard let context = CGContext( - data: &bytes, width: width, height: height, bitsPerComponent: 8, - bytesPerRow: stride, space: CGColorSpaceCreateDeviceRGB(), - bitmapInfo: CGImageAlphaInfo.premultipliedLast.rawValue - ) else { throw FeedbackError.message("Cannot create pixel context") } - context.interpolationQuality = .high - context.draw(image, in: CGRect(x: 0, y: 0, width: width, height: height)) - return (bytes, width, height) -} - -func changeRatio(baseline: URL, sample: URL, region: Bounds) throws -> Double { - let beforeImage = try decodedImage(baseline) - let afterImage = try decodedImage(sample) - guard - beforeImage.width * afterImage.height - == afterImage.width * beforeImage.height - else { - throw FeedbackError.message( - "Screenshot aspect ratio changed during hold: " - + "\(beforeImage.width)x\(beforeImage.height) to " - + "\(afterImage.width)x\(afterImage.height)" - ) - } - // Agent Device emits simulator screenshots in logical points while simctl - // emits physical pixels. Compare both at the baseline resolution when their - // aspect ratio is unchanged so capture scale is not mistaken for UI change. - let before = try rgba(beforeImage) - let after = try rgba( - afterImage, width: before.width, height: before.height - ) - // Vision's public OCR bounds define the crop. Vision uses a bottom-left - // normalized origin, while the bitmap rows are addressed from the top. - let x0 = max(0, Int(region.x * Double(before.width))) - let x1 = min(before.width, Int((region.x + region.width) * Double(before.width))) - let y0 = max(0, Int((1 - region.y - region.height) * Double(before.height))) - let y1 = min(before.height, Int((1 - region.y) * Double(before.height))) - var changed = 0, total = 0 - for y in y0..= 20 { changed += 1 } - total += 1 - } - } - guard total > 0 else { throw FeedbackError.message("Destination crop is empty") } - return Double(changed) / Double(total) -} - -func expandedLabelRegion(_ label: Label) -> Bounds { - let box = label.bounds - return Bounds( - x: max(0, box.x - box.width * 0.15), - y: max(0, box.y - box.height * 2.0), - width: min(1, box.x + box.width * 1.15) - max(0, box.x - box.width * 0.15), - height: min(1, box.y + box.height * 3.0) - max(0, box.y - box.height * 2.0) - ) -} - -func insertionBandRegion(predecessor: Label, target: Label) -> Bounds { - let predecessorCenter = predecessor.bounds.y + predecessor.bounds.height / 2 - let targetCenter = target.bounds.y + target.bounds.height / 2 - let lower = min(predecessorCenter, targetCenter) - let upper = max(predecessorCenter, targetCenter) - let left = max(0, min(predecessor.bounds.x, target.bounds.x)) - let right = min( - 1, - max( - predecessor.bounds.x + predecessor.bounds.width, - target.bounds.x + target.bounds.width - ) - ) - return Bounds(x: left, y: lower, width: right - left, height: upper - lower) -} - -func analyzeRun( - directory: URL, configuration: String, scenario: String, spec: ScenarioSpec -) throws -> Run { - let baseline = directory.appendingPathComponent("baseline.png") - guard FileManager.default.fileExists(atPath: baseline.path) else { - throw FeedbackError.message("Missing \(baseline.path)") - } - let baselineLabels = try recognize(baseline) - let resolvedVisualTargetLabel = - spec.visualTargetLabelsByConfiguration?[configuration] - ?? spec.visualTargetLabel - let destinationRegion: Bounds - if spec.kind == "drop" { - destinationRegion = expandedLabelRegion( - try exactLabel( - baselineLabels, resolvedVisualTargetLabel ?? spec.targetLabel, - path: baseline.path - ) - ) - } else { - guard let predecessorLabel = spec.predecessorLabel else { - throw FeedbackError.message("Missing predecessor label for \(scenario)") - } - destinationRegion = insertionBandRegion( - predecessor: try exactLabel(baselineLabels, predecessorLabel, path: baseline.path), - target: try exactLabel(baselineLabels, spec.targetLabel, path: baseline.path) - ) - } - var samples: [Sample] = [] - for index in 1...3 { - let url = directory.appendingPathComponent("sample-\(index).png") - guard FileManager.default.fileExists(atPath: url.path) else { - throw FeedbackError.message("Missing \(url.path)") - } - samples.append(Sample( - path: url.path, - labels: try recognize(url), - destinationChangeRatio: try changeRatio( - baseline: baseline, sample: url, region: destinationRegion - ) - )) - } - return Run( - configuration: configuration, scenario: scenario, kind: spec.kind, - sourceLabel: spec.sourceLabel, - sourceLabelAliases: spec.sourceLabelAliasesByConfiguration?[configuration], - predecessorLabel: spec.predecessorLabel, - targetLabel: spec.targetLabel, - visualTargetLabel: resolvedVisualTargetLabel, - baselineLabels: baselineLabels, - samples: samples - ) -} - -do { - if CommandLine.arguments.count == 3, CommandLine.arguments[1] == "--ocr" { - let labels = try recognize(URL(fileURLWithPath: CommandLine.arguments[2])) - let data = try JSONEncoder().encode(labels) - FileHandle.standardOutput.write(data) - FileHandle.standardOutput.write(Data("\n".utf8)) - exit(0) - } - if CommandLine.arguments.count == 7, CommandLine.arguments[1] == "--case" { - let directory = URL(fileURLWithPath: CommandLine.arguments[2]) - let specs = try JSONDecoder().decode( - Specs.self, from: Data(contentsOf: URL(fileURLWithPath: CommandLine.arguments[3])) - ) - let scenario = CommandLine.arguments[4] - guard let spec = specs.scenarios[scenario] else { - throw FeedbackError.message("Unknown scenario \(scenario)") - } - let run = try analyzeRun( - directory: directory, - configuration: CommandLine.arguments[5], - scenario: scenario, - spec: spec - ) - let data = try JSONEncoder().encode(Report(runs: [run])) - try data.write( - to: URL(fileURLWithPath: CommandLine.arguments[6]), options: .atomic - ) - exit(0) - } - guard CommandLine.arguments.count == 4 else { - throw FeedbackError.message( - "Usage: swift scripts/analyze-device-feedback.swift | --case " - ) - } - let root = URL(fileURLWithPath: CommandLine.arguments[1]) - let specs = try JSONDecoder().decode( - Specs.self, from: Data(contentsOf: URL(fileURLWithPath: CommandLine.arguments[2])) - ) - var runs: [Run] = [] - for configuration in specs.configurations { - for (scenario, spec) in specs.scenarios.sorted(by: { $0.key < $1.key }) { - let directory = root.appendingPathComponent(configuration).appendingPathComponent(scenario) - runs.append(try analyzeRun( - directory: directory, configuration: configuration, - scenario: scenario, spec: spec - )) - } - } - let data = try JSONEncoder().encode(Report(runs: runs)) - try data.write(to: URL(fileURLWithPath: CommandLine.arguments[3]), options: .atomic) -} catch { - FileHandle.standardError.write(Data("\(error)\n".utf8)) - exit(1) -} diff --git a/scripts/run-agent-device-pointer.mjs b/scripts/run-agent-device-pointer.mjs index e4b003c..942c299 100644 --- a/scripts/run-agent-device-pointer.mjs +++ b/scripts/run-agent-device-pointer.mjs @@ -35,14 +35,6 @@ if (!scenario || override?.driver !== 'agent-device') throw new Error( `No Agent Device pointer route for ${configuration}/${scenarioId}` ); -const feedbackSettleMarginMs = - override.destinationSelector.relation === 'predecessorCenter' - ? pointerTiming.predecessorCenterSettleMarginMs - : pointerTiming.agentDeviceSettleMarginMs; -const feedbackFirstSampleMs = - pointerTiming.sourceHoldMs + - pointerTiming.moveBudgetMs + - feedbackSettleMarginMs; let targetId; if (platform === 'android') { @@ -105,15 +97,10 @@ if (platform === 'android') { const sessionName = `issue39-${scenarioId}`; const sessionDeviceArgs = platform === 'ios' ? ['--udid', targetId] : []; const agentDevice = resolve('node_modules/.bin/agent-device'); -const feedbackDirectory = resolve( - process.env.ISSUE39_FEEDBACK_DIR ?? - `artifacts/issue-39/feedback/${configuration}/${scenarioId}` -); const recordingDirectory = resolve( process.env.ISSUE39_AGENT_DEVICE_DIR ?? `artifacts/issue-39/agent-device/${configuration}/${scenarioId}` ); -await mkdir(feedbackDirectory, { recursive: true }); await mkdir(recordingDirectory, { recursive: true }); const replayStateRoot = resolve( 'artifacts/issue-39/agent-device', @@ -143,18 +130,6 @@ const runSessionCommand = (...args) => [...args, '--session', sessionName, ...sessionDeviceArgs], { env: replayEnvironment } ); -const screenshot = async (path) => { - if (platform === 'android') { - const { stdout } = await execFileAsync( - 'adb', - ['-s', targetId, 'exec-out', 'screencap', '-p'], - { encoding: 'buffer', maxBuffer: 20 * 1024 * 1024 } - ); - await writeFile(path, stdout); - return; - } - await execFileAsync('xcrun', ['simctl', 'io', targetId, 'screenshot', path]); -}; const wait = (milliseconds) => new Promise((resolveWait) => setTimeout(resolveWait, milliseconds)); let replay; @@ -254,11 +229,6 @@ try { ); const replayPath = resolve(recordingDirectory, 'pointer-replay.ad'); const recordingPath = resolve(recordingDirectory, 'pointer.mp4'); - const baselinePath = resolve(feedbackDirectory, 'baseline.png'); - const gestureMarkerPath = resolve( - recordingDirectory, - 'gesture-start-marker.png' - ); const expectedLabels = scenario.expected.labels ?? [ `Current selection: ${scenario.expected.selection}`, `Callback count: ${scenario.expected.callbackCount}`, @@ -267,25 +237,16 @@ try { replayPath, agentDeviceReplayScript({ acceptDeepLinkPrompt, - baselinePath, deepLink: scenario.deepLink.replace('${ENGINE}', engine), destinationSelector, expectedLabels, - gestureMarkerPath, initialLabels: scenario.initial?.labels ?? ['Callback count: 0'], platform, recordingPath: platform === 'ios' ? undefined : recordingPath, sourceSelector, - terminalPath: resolve(feedbackDirectory, 'terminal.png'), timing: pointerTiming, }) ); - const previousBaselineMtime = await stat(baselinePath) - .then(({ mtimeMs }) => mtimeMs) - .catch(() => 0); - const previousGestureMarkerMtime = await stat(gestureMarkerPath) - .then(({ mtimeMs }) => mtimeMs) - .catch(() => 0); if (platform === 'ios') await startIosRecording(recordingPath); replay = spawn( agentDevice, @@ -302,56 +263,15 @@ try { ], { env: replayEnvironment, stdio: 'inherit' } ); - let replayExitCode; const replayExit = new Promise((resolveExit) => replay.once('exit', (code) => { replayExited = true; - replayExitCode = code; resolveExit(code); }) ); - const baselineDeadline = Date.now() + 180000; - while ( - (await stat(baselinePath) - .then(({ mtimeMs }) => mtimeMs <= previousBaselineMtime) - .catch(() => true)) && - replayExitCode == null && - Date.now() < baselineDeadline - ) { - await wait(100); - } - if (replayExitCode != null) - throw new Error( - `Agent Device replay exited ${replayExitCode} before baseline` - ); - if (Date.now() >= baselineDeadline) - throw new Error('Timed out waiting for the replay baseline marker'); - if (platform === 'ios') await writeFile(deepLinkConfirmationMarker, ''); - await screenshot(baselinePath); - const gestureMarkerDeadline = Date.now() + 60000; - while ( - (await stat(gestureMarkerPath) - .then(({ mtimeMs }) => mtimeMs <= previousGestureMarkerMtime) - .catch(() => true)) && - replayExitCode == null && - Date.now() < gestureMarkerDeadline - ) { - await wait(100); - } - if (replayExitCode != null) - throw new Error( - `Agent Device replay exited ${replayExitCode} before gesture marker` - ); - if (Date.now() >= gestureMarkerDeadline) - throw new Error('Timed out waiting for the replay gesture marker'); - await wait(feedbackFirstSampleMs); - - for (let index = 1; index <= pointerTiming.sampleCount; index += 1) { - await screenshot(resolve(feedbackDirectory, `sample-${index}.png`)); - if (index < pointerTiming.sampleCount) - await wait(pointerTiming.sampleIntervalMs); - } const replayResult = await replayExit; + if (platform === 'ios' && replayResult === 0) + await writeFile(deepLinkConfirmationMarker, ''); let snapshot; let observedOutcome; try { diff --git a/scripts/run-device-contract-isolated.mjs b/scripts/run-device-contract-isolated.mjs index b34c78d..ddcdf95 100644 --- a/scripts/run-device-contract-isolated.mjs +++ b/scripts/run-device-contract-isolated.mjs @@ -31,11 +31,6 @@ const caseOutcomeRoot = resolve( configuration, ...attemptSegments ); -const feedbackRoot = resolve( - 'artifacts/issue-39/feedback', - configuration, - ...attemptSegments -); const agentDeviceRoot = resolve( 'artifacts/issue-39/agent-device', configuration, @@ -135,7 +130,6 @@ for (const [scenarioIndex, scenario] of applicable.entries()) { env: { ...process.env, ISSUE39_AGENT_DEVICE_DIR: resolve(agentDeviceRoot, scenario.id), - ISSUE39_FEEDBACK_DIR: resolve(feedbackRoot, scenario.id), }, stdio: 'inherit', timeout: agentDeviceProcessTimeoutMs, @@ -166,7 +160,6 @@ for (const [scenarioIndex, scenario] of applicable.entries()) { ...process.env, AGENT_DEVICE_STATE_DIR: replayStateRoot, ISSUE39_DETOX_SERVER_URL: detoxServerUrlForCase(scenarioIndex), - ISSUE39_FEEDBACK_DIR: resolve(feedbackRoot, scenario.id), ISSUE39_OUTCOME_FILE: outcomeFile, }, stdio: 'inherit', diff --git a/scripts/run-device-contract-job.mjs b/scripts/run-device-contract-job.mjs index 4065c12..c2d589f 100644 --- a/scripts/run-device-contract-job.mjs +++ b/scripts/run-device-contract-job.mjs @@ -26,7 +26,7 @@ function publishDirectoryChildren(source, destination) { function publishSuccessfulAttempt(attempt) { const attemptRoot = ['attempts', attempt]; - for (const category of ['agent-device', 'feedback', 'outcomes-by-case']) + for (const category of ['agent-device', 'outcomes-by-case']) publishDirectoryChildren( resolve('artifacts/issue-39', category, configuration, ...attemptRoot), resolve('artifacts/issue-39', category, configuration) @@ -54,12 +54,7 @@ function publishSuccessfulAttempt(attempt) { } function clearPublishedEvidence() { - for (const category of [ - 'agent-device', - 'detox', - 'feedback', - 'outcomes-by-case', - ]) + for (const category of ['agent-device', 'detox', 'outcomes-by-case']) rmSync(resolve('artifacts/issue-39', category, configuration), { force: true, recursive: true, @@ -76,12 +71,7 @@ function clearPublishedEvidence() { } function clearAttemptEvidence(attempt) { - for (const category of [ - 'agent-device', - 'detox', - 'feedback', - 'outcomes-by-case', - ]) + for (const category of ['agent-device', 'detox', 'outcomes-by-case']) rmSync( resolve( 'artifacts/issue-39', diff --git a/scripts/verify-device-feedback.mjs b/scripts/verify-device-feedback.mjs deleted file mode 100644 index 0c6ffe6..0000000 --- a/scripts/verify-device-feedback.mjs +++ /dev/null @@ -1,182 +0,0 @@ -import { readFileSync } from 'node:fs'; -import { pathToFileURL } from 'node:url'; - -const MIN_SAMPLES = 3; -const MIN_DROP_CHANGE_RATIO = 0.02; -const FEEDBACK_CONFIGURATIONS = [ - 'android.fallback', - 'ios26.auto-fallback', - 'ios27.fallback', - 'ios27.native', -]; -const FEEDBACK_SCENARIOS = [ - 'free-form-reorder', - 'multi-selection-reorder', - 'scoped-drop', - 'section-list-reorder', - 'virtualized-list-reorder', -]; - -export function assertFeedbackUniverse(runs) { - const expected = FEEDBACK_CONFIGURATIONS.flatMap((configuration) => - FEEDBACK_SCENARIOS.map((scenario) => `${configuration}/${scenario}`) - ).sort(); - const actual = runs - .map(({ configuration, scenario }) => `${configuration}/${scenario}`) - .sort(); - if (new Set(actual).size !== actual.length) - throw new Error( - 'feedback report contains duplicate configuration/scenario runs' - ); - if (JSON.stringify(actual) !== JSON.stringify(expected)) - throw new Error( - `feedback report must contain exactly the canonical 20 runs; received ${actual.length}` - ); -} - -export function matchingLabels(labels, expected) { - const normalized = expected.trim().toLocaleLowerCase(); - const exact = labels.filter( - ({ text }) => text.trim().toLocaleLowerCase() === normalized - ); - if (exact.length > 0) return exact; - // Vision can confuse glyphs in a moving, semi-transparent preview - // (observed as row -> rom/roin and Card -> Sard). Normalize only those - // observed tokens; - // every other character, including the public numeric identity, stays exact. - return labels.filter(({ text }) => { - const candidate = text - .trim() - .toLocaleLowerCase() - .replace(/\bsard\b/g, 'card') - .replace(/\b(?:rom|roin)\b/g, 'row'); - return candidate === normalized; - }); -} - -function uniqueLabel(sample, expected) { - const matches = matchingLabels(sample.labels ?? [], expected); - if (matches.length !== 1) { - throw new Error( - `${sample.path}: expected exactly one public label ${JSON.stringify(expected)}, found ${matches.length}` - ); - } - return matches[0]; -} - -function optionalUniqueLabel(sample, expected) { - const matches = matchingLabels(sample.labels ?? [], expected); - if (matches.length > 1) { - throw new Error( - `${sample.path}: expected at most one public label ${JSON.stringify(expected)}, found ${matches.length}` - ); - } - return matches[0]; -} - -function centerY(label) { - return label.bounds.y + label.bounds.height / 2; -} - -export function verifyFeedbackReport(report, { enforceUniverse = true } = {}) { - if (!Array.isArray(report.runs) || report.runs.length === 0) { - throw new Error('feedback report contains no runs'); - } - if (enforceUniverse) assertFeedbackUniverse(report.runs); - for (const run of report.runs) { - if (!Array.isArray(run.samples) || run.samples.length < MIN_SAMPLES) { - throw new Error( - `${run.configuration}/${run.scenario}: expected at least ${MIN_SAMPLES} hold samples` - ); - } - let sourcePositionSamples = 0; - for (const sample of run.samples) { - if (run.kind === 'reorder') { - const baseline = { - path: `${run.configuration}/${run.scenario}/baseline.png`, - labels: run.baselineLabels, - }; - const baselinePredecessor = uniqueLabel(baseline, run.predecessorLabel); - const baselineTarget = uniqueLabel(baseline, run.targetLabel); - let sourceMatches = matchingLabels( - sample.labels ?? [], - run.sourceLabel - ); - if (sourceMatches.length === 0) { - sourceMatches = (run.sourceLabelAliases ?? []).flatMap((alias) => - matchingLabels(sample.labels ?? [], alias) - ); - } - if (sourceMatches.length > 1) { - throw new Error( - `${sample.path}: expected exactly one public label ${JSON.stringify(run.sourceLabel)}, found ${sourceMatches.length}` - ); - } - const source = sourceMatches[0]; - const predecessor = optionalUniqueLabel(sample, run.predecessorLabel); - const target = optionalUniqueLabel(sample, run.targetLabel); - const liveLabelsVisible = predecessor != null && target != null; - const lower = Math.min( - centerY(liveLabelsVisible ? predecessor : baselinePredecessor), - centerY(liveLabelsVisible ? target : baselineTarget) - ); - const upper = Math.max( - centerY(liveLabelsVisible ? predecessor : baselinePredecessor), - centerY(liveLabelsVisible ? target : baselineTarget) - ); - const sourcePosition = source == null ? null : centerY(source); - const sourcePositionValid = - sourcePosition != null && - sourcePosition > lower && - sourcePosition < upper; - if (sourcePositionValid) { - sourcePositionSamples += 1; - } - if ( - (!sourcePositionValid || !liveLabelsVisible) && - (typeof sample.destinationChangeRatio !== 'number' || - sample.destinationChangeRatio < MIN_DROP_CHANGE_RATIO) - ) { - throw new Error( - `${sample.path}: occluded insertion band visual change ${sample.destinationChangeRatio ?? 'missing'} is below ${MIN_DROP_CHANGE_RATIO}` - ); - } - } else if (run.kind === 'drop') { - uniqueLabel(sample, run.visualTargetLabel ?? run.targetLabel); - if ( - typeof sample.destinationChangeRatio !== 'number' || - sample.destinationChangeRatio < MIN_DROP_CHANGE_RATIO - ) { - throw new Error( - `${sample.path}: public destination visual change ${sample.destinationChangeRatio ?? 'missing'} is below ${MIN_DROP_CHANGE_RATIO}` - ); - } - } else { - throw new Error( - `${run.configuration}/${run.scenario}: unknown feedback kind ${run.kind}` - ); - } - } - if (run.kind === 'reorder') { - const minimumPositionSamples = Math.floor(run.samples.length / 2) + 1; - if (sourcePositionSamples < minimumPositionSamples) { - throw new Error( - `${run.configuration}/${run.scenario}: expected source-position evidence in at least ${minimumPositionSamples} of ${run.samples.length} hold samples, found ${sourcePositionSamples}` - ); - } - } - } - return report.runs.length; -} - -if (import.meta.url === pathToFileURL(process.argv[1] ?? '').href) { - const caseMode = process.argv[2] === '--case'; - const path = process.argv[caseMode ? 3 : 2]; - if (path == null) - throw new Error( - 'Usage: node scripts/verify-device-feedback.mjs [--case] ' - ); - const report = JSON.parse(readFileSync(path, 'utf8')); - const count = verifyFeedbackReport(report, { enforceUniverse: !caseMode }); - process.stdout.write(`Verified continuous feedback for ${count} runs\n`); -} diff --git a/src/__tests__/issue39-device-contract.test.ts b/src/__tests__/issue39-device-contract.test.ts index 903814d..34b90b2 100644 --- a/src/__tests__/issue39-device-contract.test.ts +++ b/src/__tests__/issue39-device-contract.test.ts @@ -323,16 +323,13 @@ describe('issue 39 portable device contract', () => { const observation = jest.requireActual<{ agentDeviceReplayScript: (options: { acceptDeepLinkPrompt: boolean; - baselinePath: string; deepLink: string; destinationSelector: string; expectedLabels: string[]; - gestureMarkerPath: string; initialLabels: string[]; platform: string; recordingPath?: string; sourceSelector: string; - terminalPath: string; timing: { destinationHoldMs: number; moveBudgetMs: number; @@ -386,16 +383,13 @@ describe('issue 39 portable device contract', () => { expect( observation.agentDeviceReplayScript({ acceptDeepLinkPrompt: false, - baselinePath: '/tmp/baseline.png', deepLink: 'reorderable://lab/free-form?engine=fallback', destinationSelector: 'label="Card row 3"', expectedLabels: ['Callback count: 1'], - gestureMarkerPath: '/tmp/gesture-start.png', initialLabels: ['Callback count: 0'], platform: 'android', recordingPath: '/tmp/pointer.mp4', sourceSelector: 'label="Card row 1"', - terminalPath: '/tmp/terminal.png', timing: { destinationHoldMs: 8000, moveBudgetMs: 1200, @@ -408,16 +402,13 @@ describe('issue 39 portable device contract', () => { expect( observation.agentDeviceReplayScript({ acceptDeepLinkPrompt: false, - baselinePath: '/tmp/baseline.png', deepLink: 'reorderable://lab/free-form?engine=fallback', destinationSelector: 'label="Card row 3"', expectedLabels: ['Callback count: 1'], - gestureMarkerPath: '/tmp/gesture-start.png', initialLabels: ['Callback count: 0'], platform: 'android', recordingPath: '/tmp/pointer.mp4', sourceSelector: 'label="Card row 1"', - terminalPath: '/tmp/terminal.png', timing: { destinationHoldMs: 8000, moveBudgetMs: 1200, @@ -425,19 +416,16 @@ describe('issue 39 portable device contract', () => { }, }) ).toContain( - 'open "${APP_TARGET}" --relaunch\nwait "Scenario Lab" 15000\nopen "${DEEP_LINK}"\nwait "Callback count: 0" 15000\nscreenshot "/tmp/baseline.png"\nrecord start "/tmp/pointer.mp4" --scope device' + 'open "${APP_TARGET}" --relaunch\nwait "Scenario Lab" 15000\nopen "${DEEP_LINK}"\nwait "Callback count: 0" 15000\nrecord start "/tmp/pointer.mp4" --scope device' ); const iosReplay = observation.agentDeviceReplayScript({ acceptDeepLinkPrompt: true, - baselinePath: '/tmp/baseline.png', deepLink: 'reorderable://lab/free-form?engine=fallback', destinationSelector: 'label="Card row 3"', expectedLabels: ['Callback count: 1'], - gestureMarkerPath: '/tmp/gesture-start.png', initialLabels: ['Callback count: 0'], platform: 'ios', sourceSelector: 'label="Card row 1"', - terminalPath: '/tmp/terminal.png', timing: { destinationHoldMs: 8000, moveBudgetMs: 1200, @@ -452,16 +440,13 @@ describe('issue 39 portable device contract', () => { expect( observation.agentDeviceReplayScript({ acceptDeepLinkPrompt: false, - baselinePath: '/tmp/baseline.png', deepLink: 'reorderable://lab/free-form?engine=fallback', destinationSelector: 'label="Card row 3"', expectedLabels: ['Callback count: 1'], - gestureMarkerPath: '/tmp/gesture-start.png', initialLabels: ['Callback count: 0'], platform: 'ios', recordingPath: '/tmp/pointer.mp4', sourceSelector: 'label="Card row 1"', - terminalPath: '/tmp/terminal.png', timing: { destinationHoldMs: 8000, moveBudgetMs: 1200, @@ -540,270 +525,41 @@ describe('issue 39 portable device contract', () => { } }); - it('verifies visible and legitimately occluded continuous feedback', () => { - const verifier = resolve(root, 'scripts/verify-device-feedback.mjs'); - const runReport = (path: string) => - execFileSync(process.execPath, [verifier, '--case', path], { - encoding: 'utf8', - }); - const runFixture = (name: string) => - runReport(resolve(root, `e2e/contracts/fixtures/${name}.json`)); - - expect(runFixture('feedback-valid')).toContain( - 'Verified continuous feedback for 2 runs' - ); - expect(runFixture('feedback-occluded-valid')).toContain( - 'Verified continuous feedback for 1 runs' - ); - expect(() => runFixture('feedback-missing')).toThrow( - /expected at least 3 hold samples/ - ); - expect(() => runFixture('feedback-ambiguous')).toThrow( - /expected exactly one public label "Card row 1", found 2/ - ); - expect(() => runFixture('feedback-baseline-ambiguous')).toThrow( - /baseline\.png: expected exactly one public label "Card row 6", found 2/ - ); - expect(() => runFixture('feedback-source-outside')).toThrow( - /expected source-position evidence in at least 2 of 3 hold samples, found 1/ - ); - expect(() => runFixture('feedback-insufficient-change')).toThrow( - /occluded insertion band visual change 0\.01 is below 0\.02/ - ); - const directory = mkdtempSync(resolve(tmpdir(), 'feedback-ocr-miss-')); - try { - const report = JSON.parse( - read('e2e/contracts/fixtures/feedback-occluded-valid.json') - ) as { - runs: Array<{ - sourceLabelAliases?: string[]; - samples: Array<{ labels: Array<{ text: string }> }>; - }>; - }; - report.runs[0]!.samples[0]!.labels = []; - const reportPath = resolve(directory, 'report.json'); - writeFileSync(reportPath, JSON.stringify(report)); - expect(runReport(reportPath)).toContain( - 'Verified continuous feedback for 1 runs' - ); - - report.runs[0]!.samples[1]!.labels = []; - writeFileSync(reportPath, JSON.stringify(report)); - expect(() => runReport(reportPath)).toThrow( - /expected source-position evidence in at least 2 of 3 hold samples, found 1/ - ); - - const aliasedReport = JSON.parse( - read('e2e/contracts/fixtures/feedback-occluded-valid.json') - ) as typeof report; - aliasedReport.runs[0]!.sourceLabelAliases = ['Card row']; - for (const sample of aliasedReport.runs[0]!.samples) { - sample.labels[0]!.text = 'Card row'; - } - writeFileSync(reportPath, JSON.stringify(aliasedReport)); - expect(runReport(reportPath)).toContain( - 'Verified continuous feedback for 1 runs' - ); - - aliasedReport.runs[0]!.samples[0]!.labels[0]!.text = 'Card row 3'; - aliasedReport.runs[0]!.samples[1]!.labels[0]!.text = 'Card row 3'; - writeFileSync(reportPath, JSON.stringify(aliasedReport)); - expect(() => runReport(reportPath)).toThrow( - /expected source-position evidence in at least 2 of 3 hold samples, found 1/ - ); - } finally { - rmSync(directory, { recursive: true }); - } - const feedbackSpecs = JSON.parse( - read('e2e/contracts/feedback-specs.json') - ) as { - scenarios: Record< - string, - { - sourceLabelAliasesByConfiguration?: Record; - targetLabel: string; - visualTargetLabel?: string; - visualTargetLabelsByConfiguration?: Record; - } - >; - }; - expect(feedbackSpecs.scenarios['scoped-drop']).toMatchObject({ - targetLabel: 'Accepting drop zone', - visualTargetLabel: 'Acceptin', - visualTargetLabelsByConfiguration: { - 'ios27.fallback': 'Drop selected items here', - 'ios26.auto-fallback': 'Drop selected items here', - 'android.fallback': 'Drop selected items here', - }, - }); - expect(feedbackSpecs.scenarios['virtualized-list-reorder']).toMatchObject({ - sourceLabelAliasesByConfiguration: { - 'android.fallback': ['List row'], - }, - }); - const analyzer = read('scripts/analyze-device-feedback.swift'); - expect(analyzer).toMatch( - /beforeImage\.width \* afterImage\.height\s*== afterImage\.width \* beforeImage\.height/ - ); - expect(analyzer).toContain( - 'afterImage, width: before.width, height: before.height' - ); - expect(analyzer).not.toContain('Screenshot dimensions changed during hold'); - expect(analyzer).toContain('trimmed.hasPrefix("\' ")'); - expect(analyzer).toContain('String(trimmed.dropFirst(2))'); - expect(analyzer).toContain( - 'sourceLabelAliases: spec.sourceLabelAliasesByConfiguration?[configuration]' - ); - }); - - it('requires exactly one feedback run for every canonical configuration/scenario pair', () => { - const verifierUrl = JSON.stringify( - `file://${resolve(root, 'scripts/verify-device-feedback.mjs')}` - ); - const result = JSON.parse( - execFileSync( - process.execPath, - [ - '--input-type=module', - '--eval', - `const { assertFeedbackUniverse } = await import(${verifierUrl}); -const configurations = ['android.fallback', 'ios26.auto-fallback', 'ios27.fallback', 'ios27.native']; -const scenarios = ['free-form-reorder', 'multi-selection-reorder', 'scoped-drop', 'section-list-reorder', 'virtualized-list-reorder']; -const runs = configurations.flatMap(configuration => scenarios.map(scenario => ({ configuration, scenario }))); -const failure = candidate => { try { assertFeedbackUniverse(candidate); return null; } catch (error) { return error.message; } }; -console.log(JSON.stringify({ - complete: failure(runs), - missing: failure(runs.slice(1)), - duplicate: failure([...runs, runs[0]]), -}));`, - ], - { encoding: 'utf8' } - ) - ); - - expect(result).toEqual({ - complete: null, - missing: - 'feedback report must contain exactly the canonical 20 runs; received 19', - duplicate: - 'feedback report contains duplicate configuration/scenario runs', - }); - }); - - it('limits moving-preview OCR tolerance to observed glyph errors without weakening identity', () => { - const verifierUrl = JSON.stringify( - `file://${resolve(root, 'scripts/verify-device-feedback.mjs')}` - ); - const result = JSON.parse( - execFileSync( - process.execPath, - [ - '--input-type=module', - '--eval', - `const { matchingLabels } = await import(${verifierUrl}); -const count = (texts, expected) => matchingLabels(texts.map((text) => ({ text })), expected).length; -console.log(JSON.stringify({ - observedTypo: count(['Card rom 2'], 'Card row 2'), - observedSplitTypo: count(['Card roin 2'], 'Card row 2'), - observedLeadTypo: count(['Sard row 2'], 'Card row 2'), - wrongNumericIdentity: count(['Card rom 3'], 'Card row 2'), - duplicateCandidates: count(['Card rom 2', 'Card rom 2'], 'Card row 2'), - missingSource: count([], 'Card row 2'), - broaderEdit: count(['Cord row 2'], 'Card row 2'), -}));`, - ], - { encoding: 'utf8' } - ) - ); - - expect(result).toEqual({ - observedTypo: 1, - observedSplitTypo: 1, - observedLeadTypo: 1, - wrongNumericIdentity: 0, - duplicateCandidates: 2, - missingSource: 0, - broaderEdit: 0, - }); - }); - - it('samples pointer feedback only after a long destination hold has settled', () => { + it('drives Agent Device pointer replays without mid-gesture screen sampling', () => { const scenarios = JSON.parse( read('e2e/contracts/scenarios.json') ) as readonly ContractScenario[]; - const feedbackSpecs = JSON.parse( - read('e2e/contracts/feedback-specs.json') - ) as { scenarios: Record }; - const pointerScenarios = scenarios.filter(({ id }) => - Object.hasOwn(feedbackSpecs.scenarios, id) - ); const timing = JSON.parse(read('e2e/contracts/pointer-timing.json')) as { - agentDeviceSettleMarginMs: number; destinationHoldMs: number; moveBudgetMs: number; - predecessorCenterSettleMarginMs: number; - sampleCount: number; - sampleIntervalMs: number; - settleMarginMs: number; sourceHoldMs: number; }; const detoxRunner = read('e2e/contracts/portable-contract.e2e.cjs'); const agentDeviceRunner = read('scripts/run-agent-device-pointer.mjs'); + // The screenshot-feedback apparatus is excised: pointer drags hold the + // destination only as long as the gesture needs, and neither runner + // samples the screen mid-gesture. + expect(Object.keys(timing).sort()).toEqual([ + 'destinationHoldMs', + 'moveBudgetMs', + 'sourceHoldMs', + ]); expect( - pointerScenarios.every( - ({ action }) => action.holdDurationMs === timing.destinationHoldMs - ) + scenarios + .filter(({ action }) => action.kind === 'drag') + .every(({ action }) => action.holdDurationMs == null) ).toBe(true); - const firstSampleMs = - timing.sourceHoldMs + timing.moveBudgetMs + timing.settleMarginMs; - const finalSampleMs = - firstSampleMs + (timing.sampleCount - 1) * timing.sampleIntervalMs; - const releaseMs = - timing.sourceHoldMs + timing.moveBudgetMs + timing.destinationHoldMs; - expect(firstSampleMs).toBeGreaterThan( - timing.sourceHoldMs + timing.moveBudgetMs - ); - expect(finalSampleMs).toBeLessThan(releaseMs); - expect(releaseMs - finalSampleMs).toBeGreaterThanOrEqual(3000); - const agentDeviceFinalSampleMs = - timing.sourceHoldMs + - timing.moveBudgetMs + - timing.agentDeviceSettleMarginMs + - (timing.sampleCount - 1) * timing.sampleIntervalMs; - expect(timing.agentDeviceSettleMarginMs).toBeGreaterThanOrEqual(3500); - expect(agentDeviceFinalSampleMs).toBeLessThan(releaseMs); - expect(releaseMs - agentDeviceFinalSampleMs).toBeGreaterThanOrEqual(3000); - const predecessorCenterFinalSampleMs = - timing.sourceHoldMs + - timing.moveBudgetMs + - timing.predecessorCenterSettleMarginMs + - (timing.sampleCount - 1) * timing.sampleIntervalMs; - expect(timing.predecessorCenterSettleMarginMs).toBeGreaterThan( - timing.agentDeviceSettleMarginMs - ); - expect(predecessorCenterFinalSampleMs).toBeLessThan(releaseMs); - expect(releaseMs - predecessorCenterFinalSampleMs).toBeGreaterThanOrEqual( - 1500 - ); expect(detoxRunner).toContain("require('./pointer-timing.json')"); - expect(detoxRunner).toContain('feedbackFirstSampleMs'); + expect(detoxRunner).not.toContain('captureOsScreenshot'); expect(agentDeviceRunner).toContain( "'../e2e/contracts/pointer-timing.json'" ); - expect(agentDeviceRunner).toContain('feedbackFirstSampleMs'); - expect(agentDeviceRunner).toContain( - 'pointerTiming.agentDeviceSettleMarginMs' - ); - expect(agentDeviceRunner).toContain( - 'pointerTiming.predecessorCenterSettleMarginMs' - ); expect(agentDeviceRunner).toContain("'replay',"); expect(agentDeviceRunner).toContain('observedLabelsFromSnapshot'); expect(agentDeviceRunner).not.toContain( '[scenarioId]: {\n labels: scenario.expected' ); - expect(agentDeviceRunner).toContain('gestureMarkerPath'); expect(agentDeviceRunner).toContain('selectorForLabel'); expect(agentDeviceRunner).toContain('`id="${testId}"`'); expect(agentDeviceRunner).toContain("'recordVideo'"); @@ -811,11 +567,10 @@ console.log(JSON.stringify({ expect(agentDeviceRunner).toContain( "recordingPath: platform === 'ios' ? undefined : recordingPath" ); - expect(agentDeviceRunner).toContain('feedbackFirstSampleMs'); - expect(agentDeviceRunner).toContain("'screencap'"); + expect(agentDeviceRunner).not.toContain('gestureMarkerPath'); + expect(agentDeviceRunner).not.toContain("'screencap'"); expect(agentDeviceRunner).toContain("'deep-link-confirmed'"); expect(agentDeviceRunner).toContain('acceptDeepLinkPrompt'); - expect(agentDeviceRunner).toContain('Date.now() + 180000'); expect(agentDeviceRunner).toMatch(/'--timeout',\s*'180000'/); expect( agentDeviceRunner.indexOf('const replayResult = await replayExit') From dfea136dfad99b83671717dfdfbec2c8e018449c Mon Sep 17 00:00:00 2001 From: Thiago Brezinski Date: Fri, 4 Sep 2026 12:26:12 +0100 Subject: [PATCH 3/8] test: add the RNTL drag render-path backstop (#67) Deterministic RNTL proof that the drag-preview overlay and the destination-feedback highlight activate during a simulated drag and deactivate at release, replacing the excised OCR/pixel screenshot-feedback gate for the broken-render-wiring regression class (ADR 0006, 2026-09-01 amendment). Mutation checks: fails when the overlay start dispatch is removed and when the destination-feedback opacity wiring is broken. Passed 10/10 under --runInBand and 10/10 under --maxWorkers=2. Co-Authored-By: Claude Fable 5 --- .../issue67-drag-render-paths.test.tsx | 131 ++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 src/__tests__/issue67-drag-render-paths.test.tsx diff --git a/src/__tests__/issue67-drag-render-paths.test.tsx b/src/__tests__/issue67-drag-render-paths.test.tsx new file mode 100644 index 0000000..457fdab --- /dev/null +++ b/src/__tests__/issue67-drag-render-paths.test.tsx @@ -0,0 +1,131 @@ +import { describe, expect, it, jest } from '@jest/globals'; +import { act, fireEvent, render } from '@testing-library/react-native'; +import { StyleSheet, Text } from 'react-native'; +import { GestureHandlerRootView } from 'react-native-gesture-handler'; +import { getByGestureTestId } from 'react-native-gesture-handler/jest-utils'; + +import { ReorderableList } from '..'; + +const items = Array.from({ length: 8 }, (_, index) => ({ + id: `item-${index}`, + label: `Item ${index}`, +})); +const getItemLayout = ( + _data: readonly (typeof items)[number][], + index: number +) => ({ index, length: 50, offset: index * 50 }); + +type ViewportHandler = { + handlers: { + onTouchesDown?: (event: { + allTouches: { absoluteY: number; x: number; y: number }[]; + }) => void; + onStart?: (event: { absoluteY: number; y: number }) => void; + onTouchesMove?: ( + event: { allTouches: { absoluteY: number; x: number; y: number }[] }, + state: { fail: () => void } + ) => void; + onEnd?: (event: { absoluteY: number; y: number }) => void; + }; +}; + +describe('issue 67 drag render-path backstop', () => { + it('activates the drag preview and destination feedback during a drag and hides both at release', async () => { + const onReorder = jest.fn(); + const view = () => ( + + item.id} + onReorder={onReorder} + renderItem={({ item }) => {item.label}} + testID="backstop-list" + /> + + ); + const rendered = await render(view()); + await fireEvent(rendered.getByTestId('backstop-list'), 'layout', { + nativeEvent: { layout: { height: 200, width: 300, x: 0, y: 0 } }, + }); + const feedbackStyle = () => + StyleSheet.flatten( + rendered.getByTestId('reorderable-list-destination-feedback', { + includeHiddenElements: true, + }).props.style + ); + const sourceCopies = () => + rendered.getAllByText('Item 0', { includeHiddenElements: true }); + const previewCopy = () => + sourceCopies().find( + ({ parent }) => parent?.props.accessibilityElementsHidden === true + ); + + expect(sourceCopies()).toHaveLength(1); + expect(previewCopy()).toBeUndefined(); + expect(feedbackStyle().opacity).toBe(0); + + const handler = getByGestureTestId( + 'reorderable-list-viewport' + ) as unknown as ViewportHandler; + // Grab item-0 (rows are 50 tall, so viewport y 25 is its middle). While + // the pointer is over the grabbed row, the destination is the following + // boundary: index 1 at content offset 50. + await act(async () => { + handler.handlers.onTouchesDown?.({ + allTouches: [{ absoluteY: 500, x: 20, y: 25 }], + }); + handler.handlers.onStart?.({ absoluteY: 500, y: 25 }); + }); + await rendered.rerender(view()); + + expect(sourceCopies()).toHaveLength(2); + const activePreview = previewCopy(); + expect(activePreview).toBeDefined(); + expect(activePreview?.parent?.props.pointerEvents).toBe('none'); + expect( + StyleSheet.flatten(activePreview?.parent?.props.style) + ).toMatchObject({ height: 50 }); + expect(feedbackStyle()).toMatchObject({ + opacity: 1, + transform: [{ translateY: 50 }], + }); + + // Move the touch down by 85 (absoluteY 500 -> 585): the pointer lands at + // viewport y 110, inside the leading half of item-2, so the insertion + // destination is index 2 and the feedback sits on its boundary at 100. + await act(async () => { + handler.handlers.onTouchesMove?.( + { allTouches: [{ absoluteY: 585, x: 20, y: 0 }] }, + { fail: jest.fn() } + ); + }); + await rendered.rerender(view()); + + expect(feedbackStyle()).toMatchObject({ + opacity: 1, + transform: [{ translateY: 100 }], + }); + const movedPreviewStyle = StyleSheet.flatten( + previewCopy()?.parent?.props.style + ); + expect(movedPreviewStyle).toBeDefined(); + // The preview tracks the pointer translation (85) plus the iOS active + // preview visual offset of -12: 0 (origin) + 85 - 12 = 73. + expect(movedPreviewStyle?.transform).toEqual([{ translateY: 73 }]); + + await act(async () => handler.handlers.onEnd?.({ absoluteY: 585, y: 110 })); + expect(onReorder).toHaveBeenCalledWith( + expect.objectContaining({ + sourceIds: ['item-0'], + destination: { sectionId: null, beforeId: 'item-2' }, + }) + ); + await rendered.rerender(view()); + + expect(sourceCopies()).toHaveLength(1); + expect(previewCopy()).toBeUndefined(); + expect(feedbackStyle().opacity).toBe(0); + await rendered.unmount(); + }); +}); From 48642044768d5fb1b279bed8c05ab3f8d063609b Mon Sep 17 00:00:00 2001 From: Thiago Brezinski Date: Fri, 4 Sep 2026 22:03:16 +0100 Subject: [PATCH 4/8] test(ci): delete the YAML-regex change-detector tests, keep the script-execution tests (#66) Drop the 8 tests that asserted against workflow YAML and package.json content -- they failed whenever the YAML was edited, not when it was wrong, and the CI overhaul is all edits. The 3 script-execution tests survive in scripts/__tests__/consumer-scripts.test.mjs, renamed to match its remaining content. Each invariant that survives the gate map is preserved as a comment beside the line it guards: per-event concurrency groups (ci.yml, docs.yml; exact-package-candidate.yml already had one), detox build-framework-cache ordering, Gradle invoked from the generated consumer directory, and the Jest self-reference mapping in package.json. Co-Authored-By: Claude Fable 5 --- .github/workflows/ci.yml | 2 + .github/workflows/docs.yml | 2 + .github/workflows/exact-package-candidate.yml | 5 + package.json | 1 + scripts/__tests__/ci-fixtures.test.mjs | 223 ------------------ scripts/__tests__/consumer-scripts.test.mjs | 99 ++++++++ 6 files changed, 109 insertions(+), 223 deletions(-) delete mode 100644 scripts/__tests__/ci-fixtures.test.mjs create mode 100644 scripts/__tests__/consumer-scripts.test.mjs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 20df82a..f964656 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -11,6 +11,8 @@ on: - checks_requested concurrency: + # Keep the event name in the group: sharing one group across events once let + # a release-authorizing push run deadlock behind other events for 13 hours. group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }} cancel-in-progress: true diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 199aa93..a6abfd0 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -13,6 +13,8 @@ permissions: contents: read concurrency: + # Keep the event name in the group: sharing one group across events once let + # a release-authorizing push run deadlock behind other events for 13 hours. group: docs-${{ github.event_name }}-${{ github.ref }} cancel-in-progress: true diff --git a/.github/workflows/exact-package-candidate.yml b/.github/workflows/exact-package-candidate.yml index 48daed7..ffa11d5 100644 --- a/.github/workflows/exact-package-candidate.yml +++ b/.github/workflows/exact-package-candidate.yml @@ -439,6 +439,8 @@ jobs: target: google_apis profile: pixel_2 script: | + # Gradle must run from the generated consumer directory, not the + # repository root, or it builds the wrong project. cd "$RUNNER_TEMP/ReorderableConsumer/android" && ./gradlew app:installDebug --no-daemon -PnewArchEnabled=true -PhermesEnabled=true adb shell am start -W -n com.reorderableconsumer/.MainActivity @@ -576,6 +578,9 @@ jobs: bundle install bundle exec pod install --project-directory=ios ) + # build-framework-cache must run before detox build: without a + # prebuilt framework cache, Detox compiles it lazily mid-build and + # the Apple contract build fails on hosted runners. yarn detox build-framework-cache yarn detox build --configuration '${{ matrix.configuration }}' diff --git a/package.json b/package.json index c6c17e3..5ae9b7e 100644 --- a/package.json +++ b/package.json @@ -219,6 +219,7 @@ "/example/node_modules", "/lib/" ], + "//": "The self-reference mapping below (with the react-native-reorderable-source export condition above) makes Jest resolve the package name to src/ in a clean checkout; without it, tests require a build in lib/.", "moduleNameMapper": { "^react-native-reorderable$": "/src/index.tsx", "^react-native-reorderable/(flash-list|legend-list)$": "/src/$1.tsx" diff --git a/scripts/__tests__/ci-fixtures.test.mjs b/scripts/__tests__/ci-fixtures.test.mjs deleted file mode 100644 index 641fa34..0000000 --- a/scripts/__tests__/ci-fixtures.test.mjs +++ /dev/null @@ -1,223 +0,0 @@ -import assert from 'node:assert/strict'; -import { execFileSync } from 'node:child_process'; -import { - chmodSync, - mkdtempSync, - readFileSync, - statSync, - writeFileSync, -} from 'node:fs'; -import { tmpdir } from 'node:os'; -import { resolve } from 'node:path'; -import test from 'node:test'; - -const repository = resolve(import.meta.dirname, '../..'); - -test('Jest resolves the package self-reference to source in a clean checkout', () => { - const metadata = JSON.parse( - readFileSync(resolve(repository, 'package.json'), 'utf8') - ); - - assert.ok( - metadata.jest.testEnvironmentOptions.customExportConditions.includes( - 'react-native-reorderable-source' - ) - ); - assert.equal( - metadata.jest.moduleNameMapper['^react-native-reorderable$'], - '/src/index.tsx' - ); -}); - -test('Scenario Lab permits the candidate tarball to update the temporary lockfile', () => { - const workflow = readFileSync( - resolve(repository, '.github/workflows/exact-package-candidate.yml'), - 'utf8' - ); - - assert.match( - workflow, - /install-candidate-in-example\.mjs[\s\S]*?yarn install --no-immutable/ - ); - assert.match( - workflow, - /npx pod-install[\s\S]*?patch-fmt-consteval\.mjs[\s\S]*?xcodebuild/ - ); - assert.match( - workflow, - /expo-modules-autolinking react-native-config[\s\S]*?--platform '\$\{\{ matrix\.platform \}\}'[\s\S]*?--json/ - ); - assert.match( - workflow, - /EVENT_NAME: \$\{\{ github\.event_name \}\}[\s\S]*?PUSH_BEFORE_SHA: \$\{\{ github\.event\.before \}\}[\s\S]*?sha=\$PUSH_BEFORE_SHA/ - ); -}); - -test('root-level Apple workflows run CocoaPods from the bare example', () => { - for (const workflowPath of [ - '.github/workflows/exact-package-candidate.yml', - '.github/workflows/physical-performance.yml', - ]) { - const workflow = readFileSync(resolve(repository, workflowPath), 'utf8'); - assert.match( - workflow, - /cd example[\s\S]*?bundle install[\s\S]*?bundle exec pod install --project-directory=ios/ - ); - } -}); - -test('release-authorizing push gates do not share concurrency with other events', () => { - for (const workflowPath of [ - '.github/workflows/ci.yml', - '.github/workflows/docs.yml', - '.github/workflows/exact-package-candidate.yml', - ]) { - const workflow = readFileSync(resolve(repository, workflowPath), 'utf8'); - - assert.match( - workflow, - /group: .*\$\{\{ github\.event_name \}\}.*\$\{\{ github\.ref \}\}/, - workflowPath - ); - } -}); - -test('Apple contract jobs prepare the Detox framework cache explicitly', () => { - const workflow = readFileSync( - resolve(repository, '.github/workflows/exact-package-candidate.yml'), - 'utf8' - ); - - assert.match( - workflow, - /yarn detox build-framework-cache[\s\S]*?yarn detox build --configuration/ - ); -}); - -test('manual Reassure runs compare against a prior commit', () => { - const workflow = readFileSync( - resolve(repository, '.github/workflows/exact-package-candidate.yml'), - 'utf8' - ); - - assert.match( - workflow, - /baseline_sha="\$\(git merge-base HEAD origin\/main\)"[\s\S]*?if \[\[ "\$baseline_sha" == "\$\(git rev-parse HEAD\)" \]\]; then[\s\S]*?baseline_sha="\$\(git rev-parse HEAD\^\)"/ - ); - assert.doesNotMatch( - workflow, - /echo "sha=\$\(git merge-base HEAD origin\/main\)"/ - ); -}); - -test('the minimum Android runtime invokes Gradle from its generated fixture', () => { - const workflow = readFileSync( - resolve(repository, '.github/workflows/exact-package-candidate.yml'), - 'utf8' - ); - - assert.match( - workflow, - /cd "\$RUNNER_TEMP\/ReorderableConsumer\/android" && \.\/gradlew app:installDebug/ - ); -}); - -test('the iOS floor check uses the oldest durable hosted runtime', () => { - const workflow = readFileSync( - resolve(repository, '.github/workflows/exact-package-candidate.yml'), - 'utf8' - ); - - assert.match( - workflow, - /minimum-ios-runtime:[\s\S]*?name: Nightly iOS 15\.1 deployment target on hosted iOS 18\.5[\s\S]*?runs-on: macos-15/ - ); - assert.match(workflow, /IPHONEOS_DEPLOYMENT_TARGET = 15\\\.1/); - assert.match(workflow, /SimRuntime\.iOS-18-5/); - assert.doesNotMatch(workflow, /runs-on: \[self-hosted, macOS, ios-15\.1\]/); -}); - -test('Expo clean consumers do not install the React Native community CLI', () => { - const fixture = mkdtempSync(resolve(tmpdir(), 'reorderable-consumer-')); - writeFileSync( - resolve(fixture, 'package.json'), - `${JSON.stringify({ dependencies: {} }, null, 2)}\n` - ); - - execFileSync( - process.execPath, - [ - resolve(repository, 'scripts/configure-clean-consumer.mjs'), - fixture, - resolve(repository, 'candidate.tgz'), - 'expo', - JSON.stringify({ - react: '19.2.3', - reactNative: '0.86.2', - reactNativeGestureHandler: '3.2.1', - reactNativeReanimated: '4.5.3', - reactNativeWorklets: '0.11.4', - }), - ], - { stdio: 'pipe' } - ); - - const metadata = JSON.parse( - readFileSync(resolve(fixture, 'package.json'), 'utf8') - ); - assert.equal( - metadata.devDependencies?.['@react-native-community/cli'], - undefined - ); -}); - -test('the autolinking verifier accepts Expo platform-specific output', () => { - execFileSync( - process.execPath, - [ - resolve(repository, 'scripts/verify-consumer-autolinking.mjs'), - '--platform', - 'ios', - ], - { - input: JSON.stringify({ - dependencies: { - 'react-native-reorderable': { - platforms: { ios: { podspecPath: 'Reorderable.podspec' } }, - }, - }, - }), - stdio: ['pipe', 'pipe', 'pipe'], - } - ); -}); - -test('the fmt workaround disables only the vulnerable 11.0.2 consteval branch', () => { - const fixture = mkdtempSync(resolve(tmpdir(), 'reorderable-fmt-')); - const header = resolve(fixture, 'base.h'); - writeFileSync( - header, - `#define FMT_VERSION 110002 -#elif defined(__cpp_consteval) -# define FMT_USE_CONSTEVAL 1 -#elif FMT_GCC_VERSION >= 1002 || FMT_CLANG_VERSION >= 1101 -# define FMT_USE_CONSTEVAL 1 -` - ); - chmodSync(header, 0o444); - - const patcher = resolve(repository, 'scripts/patch-fmt-consteval.mjs'); - execFileSync(process.execPath, [patcher, header]); - execFileSync(process.execPath, [patcher, header]); - - const patched = readFileSync(header, 'utf8'); - assert.match( - patched, - /#elif defined\(__cpp_consteval\)\n# define FMT_USE_CONSTEVAL 0/ - ); - assert.match( - patched, - /#elif FMT_GCC_VERSION >= 1002 \|\| FMT_CLANG_VERSION >= 1101\n# define FMT_USE_CONSTEVAL 1/ - ); - assert.equal(statSync(header).mode & 0o777, 0o444); -}); diff --git a/scripts/__tests__/consumer-scripts.test.mjs b/scripts/__tests__/consumer-scripts.test.mjs new file mode 100644 index 0000000..ee13648 --- /dev/null +++ b/scripts/__tests__/consumer-scripts.test.mjs @@ -0,0 +1,99 @@ +import assert from 'node:assert/strict'; +import { execFileSync } from 'node:child_process'; +import { + chmodSync, + mkdtempSync, + readFileSync, + statSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { resolve } from 'node:path'; +import test from 'node:test'; + +const repository = resolve(import.meta.dirname, '../..'); + +test('Expo clean consumers do not install the React Native community CLI', () => { + const fixture = mkdtempSync(resolve(tmpdir(), 'reorderable-consumer-')); + writeFileSync( + resolve(fixture, 'package.json'), + `${JSON.stringify({ dependencies: {} }, null, 2)}\n` + ); + + execFileSync( + process.execPath, + [ + resolve(repository, 'scripts/configure-clean-consumer.mjs'), + fixture, + resolve(repository, 'candidate.tgz'), + 'expo', + JSON.stringify({ + react: '19.2.3', + reactNative: '0.86.2', + reactNativeGestureHandler: '3.2.1', + reactNativeReanimated: '4.5.3', + reactNativeWorklets: '0.11.4', + }), + ], + { stdio: 'pipe' } + ); + + const metadata = JSON.parse( + readFileSync(resolve(fixture, 'package.json'), 'utf8') + ); + assert.equal( + metadata.devDependencies?.['@react-native-community/cli'], + undefined + ); +}); + +test('the autolinking verifier accepts Expo platform-specific output', () => { + execFileSync( + process.execPath, + [ + resolve(repository, 'scripts/verify-consumer-autolinking.mjs'), + '--platform', + 'ios', + ], + { + input: JSON.stringify({ + dependencies: { + 'react-native-reorderable': { + platforms: { ios: { podspecPath: 'Reorderable.podspec' } }, + }, + }, + }), + stdio: ['pipe', 'pipe', 'pipe'], + } + ); +}); + +test('the fmt workaround disables only the vulnerable 11.0.2 consteval branch', () => { + const fixture = mkdtempSync(resolve(tmpdir(), 'reorderable-fmt-')); + const header = resolve(fixture, 'base.h'); + writeFileSync( + header, + `#define FMT_VERSION 110002 +#elif defined(__cpp_consteval) +# define FMT_USE_CONSTEVAL 1 +#elif FMT_GCC_VERSION >= 1002 || FMT_CLANG_VERSION >= 1101 +# define FMT_USE_CONSTEVAL 1 +` + ); + chmodSync(header, 0o444); + + const patcher = resolve(repository, 'scripts/patch-fmt-consteval.mjs'); + execFileSync(process.execPath, [patcher, header]); + execFileSync(process.execPath, [patcher, header]); + + const patched = readFileSync(header, 'utf8'); + assert.match( + patched, + /#elif defined\(__cpp_consteval\)\n# define FMT_USE_CONSTEVAL 0/ + ); + assert.match( + patched, + /#elif FMT_GCC_VERSION >= 1002 \|\| FMT_CLANG_VERSION >= 1101\n# define FMT_USE_CONSTEVAL 1/ + ); + assert.equal(statSync(header).mode & 0o777, 0o444); +}); From fe6f4544073df749508a3859aa4816f84009173d Mon Sep 17 00:00:00 2001 From: Thiago Brezinski Date: Fri, 4 Sep 2026 22:48:35 +0100 Subject: [PATCH 5/8] test: drop the workflow-YAML assertions from the issue-39 device-contract suite (#66) Follow-up to 4864204: issue39-device-contract.test.ts also read exact-package-candidate.yml and asserted against its text (matrix configuration names, timeout-minutes, api-level, profile, retry flags, job-runner path). Same change-detector failure mode the issue removed from ci-fixtures.test.mjs -- they trip when the workflow is edited, not when it is wrong, and the remaining overhaul tasks are all edits. The script/config assertions in those tests stay. Co-Authored-By: Claude Fable 5 --- src/__tests__/issue39-device-contract.test.ts | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/src/__tests__/issue39-device-contract.test.ts b/src/__tests__/issue39-device-contract.test.ts index 34b90b2..ed01f9b 100644 --- a/src/__tests__/issue39-device-contract.test.ts +++ b/src/__tests__/issue39-device-contract.test.ts @@ -625,7 +625,6 @@ describe('issue 39 portable device contract', () => { it('pins the exact four engine/runtime configurations and retains recordings', () => { const config = read('.detoxrc.cjs'); - const workflow = read('.github/workflows/exact-package-candidate.yml'); const isolatedRunner = read('scripts/run-device-contract-isolated.mjs'); const jobRunner = read('scripts/run-device-contract-job.mjs'); @@ -636,7 +635,6 @@ describe('issue 39 portable device contract', () => { 'android.fallback', ]) { expect(config).toContain(`'${configuration}'`); - expect(workflow).toContain(configuration); } expect(read('package.json')).toContain('"detox": "20.51.4"'); expect(read('e2e/jest.config.cjs')).toContain( @@ -660,8 +658,6 @@ describe('issue 39 portable device contract', () => { ); expect(isolatedRunner).toContain("'--record-videos'"); expect(isolatedRunner).toContain("'--take-screenshots'"); - expect(workflow).toContain('if: always()'); - expect(workflow).not.toContain('--retries'); expect(jobRunner).toContain('index <= 2'); expect(jobRunner).toContain('ISSUE39_MATRIX_ATTEMPT: attempt'); expect(jobRunner).toContain('publishSuccessfulAttempt(attempt)'); @@ -670,10 +666,6 @@ describe('issue 39 portable device contract', () => { expect(isolatedRunner).toContain('reachedContractAssertion'); expect(isolatedRunner).toContain("failureKind === 'infrastructure'"); expect(isolatedRunner).toContain('infrastructureFailure ? 75 : 1'); - expect(workflow).toContain('timeout-minutes: 150'); - expect(workflow).toContain('api-level: 36'); - expect(workflow).toContain('profile: pixel_2'); - expect(workflow).not.toContain('profile: pixel_7_pro'); }); it('keeps the physical iOS 27 VoiceOver record tied to the named public contract', () => { @@ -725,7 +717,6 @@ describe('issue 39 portable device contract', () => { const jobRunner = read('scripts/run-device-contract-job.mjs'); const iosReset = read('scripts/reset-device-contract-ios-simulator.mjs'); const detoxConfig = read('.detoxrc.cjs'); - const workflow = read('.github/workflows/exact-package-candidate.yml'); expect(runner.indexOf('onActionStarted?.();')).toBeLessThan( runner.indexOf('await actionStarted;') @@ -813,6 +804,5 @@ describe('issue 39 portable device contract', () => { expect(iosReset).toContain("['simctl', 'boot', target.udid]"); expect(iosReset).toContain("['simctl', 'bootstatus', target.udid, '-b']"); expect(iosReset).not.toContain("['simctl', 'shutdown', 'all']"); - expect(workflow).toContain('scripts/run-device-contract-job.mjs'); }); }); From 763fa06fb3bc23e9d4c668f8f9e5794a87e1e718 Mon Sep 17 00:00:00 2001 From: Thiago Brezinski Date: Fri, 4 Sep 2026 23:11:38 +0100 Subject: [PATCH 6/8] test(ci): drop the jest comment key from package.json (#66) JSON cannot carry a real comment and the "//" convention reads as an accident in review; the Jest self-reference invariant stays documented by the mapping itself. Co-Authored-By: Claude Fable 5 --- package.json | 1 - 1 file changed, 1 deletion(-) diff --git a/package.json b/package.json index 5ae9b7e..c6c17e3 100644 --- a/package.json +++ b/package.json @@ -219,7 +219,6 @@ "/example/node_modules", "/lib/" ], - "//": "The self-reference mapping below (with the react-native-reorderable-source export condition above) makes Jest resolve the package name to src/ in a clean checkout; without it, tests require a build in lib/.", "moduleNameMapper": { "^react-native-reorderable$": "/src/index.tsx", "^react-native-reorderable/(flash-list|legend-list)$": "/src/$1.tsx" From 6744a9271088774362fb7262c610bd10d2f5244d Mon Sep 17 00:00:00 2001 From: Thiago Brezinski Date: Sat, 5 Sep 2026 08:21:26 +0100 Subject: [PATCH 7/8] ci(release): delete the gh run list gate from the mode job (#69) The main ruleset (PRs required, strict up-to-date checks, zero bypass actors) is the authorization that every main SHA passed the fast set; the sha256-pinned tarball chain remains the only runtime verification. ADR 0010 amended to record the move. Co-Authored-By: Claude Fable 5 --- .github/workflows/release.yml | 16 ---------------- ...e-documentation-and-contribution-lifecycle.md | 2 ++ 2 files changed, 2 insertions(+), 16 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 74040ea..4db8598 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -32,22 +32,6 @@ jobs: - name: Setup uses: ./.github/actions/setup - - name: Require ordinary push gates for the same commit - env: - GH_TOKEN: ${{ github.token }} - RELEASE_COMMIT: ${{ github.event.workflow_run.head_sha }} - run: | - for workflow in ci.yml docs.yml; do - conclusion=$(gh run list \ - --workflow "$workflow" \ - --commit "$RELEASE_COMMIT" \ - --event push \ - --limit 10 \ - --json conclusion,status \ - --jq '[.[] | select(.status == "completed")][0].conclusion') - test "$conclusion" = success - done - - name: Select Changesets mode id: mode run: node scripts/select-release-mode.mjs diff --git a/docs/adr/0010-release-documentation-and-contribution-lifecycle.md b/docs/adr/0010-release-documentation-and-contribution-lifecycle.md index 85012b1..b2beb55 100644 --- a/docs/adr/0010-release-documentation-and-contribution-lifecycle.md +++ b/docs/adr/0010-release-documentation-and-contribution-lifecycle.md @@ -3,3 +3,5 @@ The first stable release is `1.0.0`, not `0.x`, because the v1 effort freezes the exported API and portable behaviour and therefore makes a SemVer stability promise. Releases accumulate through Changesets and are cut on demand from an automated release PR; CI packs once, proves the exact tarball according to ADR-0006, and publishes that artifact through npm trusted publishing, while an optional `next` channel carries prereleases. Rspress on GitHub Pages is the canonical documentation surface, backed by repository-authored guides, generated TypeScript API reference, and type-checked examples; the README remains a quick start. Release-surface pull requests require a changeset and the relevant full CI matrix unless a maintainer records a commit-scoped `No changeset needed` override, while contributor-facing guidance stays separate from maintainer-only agent and triage conventions. The project publishes its compatibility matrix but makes no maintenance cadence, SLA, continuity, or backport promise and adds no separate security policy for v1. + +Amended 2026-09-05 ([#58](https://github.com/thiagobrez/react-native-reorderable/issues/58)): fast-check authorization moved from runtime re-verification to the `main` ruleset. The release workflow no longer polls `gh run list` for ci.yml/docs.yml conclusions on the release commit; the ruleset (pull requests required, strict up-to-date checks, zero bypass actors) is the guarantee that every `main` SHA passed the fast set. The sha256-pinned tarball chain — `workflow_run` gated to `event == push && head_branch == main`, artifact download pinned to the triggering run id, and `publish-verified-candidate.mjs` re-asserting sha256/byte-length/sourceCommit — is the only runtime verification. From cc00f73ecdcdceeacf3148503ce5bae4fd784363 Mon Sep 17 00:00:00 2001 From: Thiago Brezinski Date: Sat, 5 Sep 2026 08:23:48 +0100 Subject: [PATCH 8/8] docs(releasing): drop the same-commit CI/docs green requirement (#69) The gh run list gate that enforced it was deleted; the main ruleset's required pull-request checks now carry that guarantee. Co-Authored-By: Claude Fable 5 --- docs/releasing.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/releasing.md b/docs/releasing.md index b11a0d8..6bce818 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -87,7 +87,7 @@ The release workflow intentionally has no npm token. Its publish job receives a Every user-visible package change carries a file from `yarn changeset`. After a fully green push to `main`, Changesets creates or updates `changeset-release/main`. This pull request is the on-demand release boundary: leave it open while accumulating changes and merge it when the release should happen. -Merging the release pull request causes the exact-package workflow to build one tarball and run that same tarball through package inspection, clean consumers, the supported React Native compatibility matrix, performance checks, and the four-engine device contract. Publication starts only after that workflow and the ordinary `CI` and `Documentation` workflows are green for the same commit. Immediately before npm, the publisher rechecks the tarball bytes, SHA-256, source commit, package name, and version against its manifest. +Merging the release pull request causes the exact-package workflow to build one tarball and run that same tarball through package inspection, clean consumers, the supported React Native compatibility matrix, performance checks, and the four-engine device contract. Publication starts only after that workflow succeeds; the `main` ruleset's required pull-request checks are what guarantee the commit already passed the fast set. Immediately before npm, the publisher rechecks the tarball bytes, SHA-256, source commit, package name, and version against its manifest. Changesets then creates the package tag and GitHub release. The same release workflow builds and deploys the documentation from that tagged commit. Stable versions use the npm `latest` tag.