diff --git a/CHANGELOG.md b/CHANGELOG.md index fe6913a..ec976f9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,8 +7,65 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] — 0.6.0 +QR passthrough, and typed errors on everything `raiseHand` throws. + +A device-change check draws a QR code and asks for a phone — and handraise's +human is holding that phone, looking at the code on its screen. A phone cannot +scan itself, so until now this needed a second device. The agent reads the code +off the page instead and hands the human the link. + +The QR half is additive: no existing call, type or outcome changes. The errors +half is not entirely — a page that is already dead now throws instead of +returning `disconnected`. Both are under **Changed**. + ### Added +- **A `Scan QR` key on the phone, in takeover mode.** It asks the agent to read + the QR codes on the page; the answer arrives as a sheet showing what each one + said, with **Open in new tab** and **Copy**. The button is disabled while a + scan is in flight and releases itself if no answer comes. +- **Two protocol messages**: human→agent `{ type: "scanqr" }`, and agent→human + `{ type: "links", links: ScannedLink[], source: "qr" }` — always sent, with + an empty list when nothing decoded, because silence reads as a broken button. + The relay routes `scanqr` in takeover mode only. +- **`scanQrLinks(png)`, `createQrScanner()` and `OPENABLE_SCHEMES` are + exported.** The decoder is usable without a human: hand it a PNG screenshot, + get back up to two `{ text, kind }`. `scanQrLinks` is synchronous; + `createQrScanner()` is the same decode on a worker thread, which is what a + handoff uses — measured, a 4K screenshot held the event loop for 2132 ms and + now holds it for 1 ms. +- **`kind` is `"url"` only for `http:`, `https:` and `mailto:`**, with no + control or bidi characters and no credentials in the authority. `tel:` and + `otpauth:` were openable in earlier drafts of this release and are not: a + dialler control sequence and an authenticator enrolment are actions rather + than pages, and neither is worth one tap from a page nobody vetted. They are + still decoded, shown in full and copyable, under a label that names them. +- **The phone applies that whole rule again** rather than trusting a label that + crossed a socket a stranger holding the link can write to, and an openable + link is displayed, anchored and copied as the address it resolves to, with + the host as the loud part — so a homograph host cannot show one address and + open another. +- **The PNG decoder refuses what it cannot have produced.** Dimensions, + compressed size and the exact inflated length are all bounded by the header + before a byte is decompressed, chunk boundaries are walked, and IEND is + required. CRCs are deliberately not checked, and the ADR says why. +- **The relay bounds its own ingress**: 4 KiB per human message enforced before + the parse, the scan floor enforced there as well as in the core, and the + human socket held while the agent's is backpressured — with terminal answers + delivered first, never dropped. +- **`HandoffEvent.qrScans` and `HandoffEvent.qrHits`.** Two new **required** + number fields on the wide event — additive for callers, who receive the + event rather than construct it, but a TypeScript consumer that builds a + `HandoffEvent` literal in a test will need them. Both are 0 in approval mode, + which offers no scan. `qrScans - qrHits` is the number worth watching. +- **`jsqr` as a runtime dependency** (pure JavaScript, no dependencies of its + own), and a PNG decoder written against `node:zlib` in `src/core/png.ts`. + `BarcodeDetector` does not exist in Solari's Chromium, so the decode happens + in the agent process — never in the remote page, whose JavaScript belongs to + whoever the agent got stuck on. +- **[ADR 0008](docs/adr/0008-qr-passthrough.md)** and + **[measurement 05](docs/measurements/05-qr.md)**, reproducible with + `bun --env-file=.env scripts/measure-qr-decode.ts`. - **Typed errors: `HandraiseError`, `HandraiseErrorCode`, `isHandraiseError`.** Everything `raiseHand` throws now carries a `code` you can branch on — `missing_api_key`, `invalid_mode`, `empty_action`, `browser_unusable`, @@ -59,6 +116,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 `node20`). It always was the floor: `@solarisdk/browser` and the patchright runtime it wraps require Node 20, so a Node 18 install never worked; the package just did not say so. +- **`src/relay/guest/server.js` names its wire vocabulary.** `MSG` and `MODE` + replace the bare strings the untyped relay compared against, and the mobile + page it serves is handed the same object at serve time instead of keeping its + own copy. `relay.test.ts` asserts `MSG` against the TypeScript protocol's own + unions, so neither side can grow a message alone. No behaviour change. - **A page that is already dead is now refused instead of handed off.** `raiseHand` used to create a relay, fail on the first CDP call and *return* `{ outcome: "disconnected" }`. It now throws a `HandraiseError` @@ -83,6 +145,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 Apart from the page check above, nothing throws that did not throw before, and no outcome became an exception. +### Known limits + +- A scan takes a fresh full-resolution `page.screenshot()`, 293 ms p50 measured + from Germany. It is rate-limited to one per 2 s in the core. +- A symbol drawn below about 120 CSS pixels does not decode. The live cast + frame — 800 px, JPEG quality 60 — fails well before that, which is why the + scan does not reuse it. +- A code the page drew at a resampled size can be sharp and still not be found + on the first pass, so a scan looks again at 2x and then at four overlapping + corners. A page with no code at all pays all three, about 320 ms of CPU. +- Two codes on one screen need a tiled second pass to be found at all; three or + more are not attempted. +- **reCAPTCHA itself is untested.** Its demo never served the scan-to-verify + variant, which Google shows at its own discretion. The mechanism is proven + end to end in the live e2e against a page that behaves the same way. + ## [0.5.1] - 2026-09-02 Republish of 0.5.0 with no code change. 0.5.0 was published to npm and diff --git a/README.md b/README.md index 536b567..96ffbb9 100644 --- a/README.md +++ b/README.md @@ -206,8 +206,8 @@ zoom and pan yourself; double-tap toggles between zoomed and fit. Typing goes straight into the focused field, character by character — and if that field is a one-time code, the phone offers the SMS code it just received. -Four keys under the input, because a phone's virtual keyboard cannot be trusted -to send them: +Under the input, four keys a phone's virtual keyboard cannot be trusted to +send, and one that asks the agent a question about the page: | Key | What it does | |---|---| @@ -215,6 +215,44 @@ to send them: | ⇥ | Move to the next field | | ⏎ | Submit / press Enter | | Clear | Empty the focused field (select-all + backspace; disabled while nothing is focused, and kept well away from ⌫) | +| Scan QR | Read the QR codes on the page and show what they say | + +### QR codes on the page + +Some walls ask for a *second device*: reCAPTCHA's "scan to verify", a WhatsApp +Web login, an authenticator enrolment. The human is holding the phone the site +wants — and the code is on that phone's screen, so it cannot be scanned. + +**Scan QR** asks the agent instead. It takes a fresh full-resolution screenshot +of the page, decodes it, and sends back what each code said. The phone shows the +link in full and offers **Open in new tab** — so the link is opened on the +phone, which is the device the site was asking for. Takeover mode only, one scan +per 2 seconds, and a symbol below about 120 CSS pixels will not decode — scroll +or zoom the remote page and scan again. + +The code came off a page nobody vetted, so: + +- Only `http`, `https` and `mailto:` get an **Open** button. Everything else — + `javascript:`, `data:`, `blob:`, `content:`, and anything carrying an + invisible character — is shown as text with a Copy button and no link. The + agent classifies it and the phone applies the same rule again, because the + handoff URL is a bearer credential and the socket behind it takes messages + from anyone holding it. +- **`tel:` and `otpauth:` are shown and copyable, never opened.** A `tel:` code + can carry a dialler control sequence, and an `otpauth:` code enrols a TOTP + secret in your authenticator. Both are one tap and hard to take back, so the + sheet names them ("Phone number", "Authenticator secret") and you hand them + to the right app yourself. +- An openable link is shown **as the address it opens**, with the host as the + loud part of it. `https://аpple.com` with a Cyrillic а reads as apple.com and + goes to `xn--pple-43d.com`; the sheet shows the second one and says the code + wrote it differently. +- The agent process never fetches any of it, and never decodes on its own event + loop: the decode runs on a worker thread, so a handoff stays answerable while + it happens. + +Measured: [`docs/measurements/05-qr.md`](docs/measurements/05-qr.md); the +decisions are in [ADR 0008](docs/adr/0008-qr-passthrough.md). Below that, two ways out. **✋ Hand back** ends the handoff as `resolved` — one tap, the agent continues. **I can't do this** ends it as `aborted` — the agent @@ -289,6 +327,21 @@ await raiseHand(page, { | `timeout` | both | Nobody answered within `timeoutMs`. | | `disconnected` | both | The browser session died mid-handoff. | +### `scanQrLinks(png): ScannedLink[]` + +The decoder behind the phone's **Scan QR** button, exported so an agent can +read a code without asking a human. Takes the bytes of a PNG screenshot, +returns up to two `{ text, kind }` — `kind: "url"` only for a scheme in +`OPENABLE_SCHEMES`, which is also exported. It reads and classifies; it never +opens anything. + +```ts +import { scanQrLinks } from "handraise" + +const codes = scanQrLinks(await page.screenshot({ type: "png" })) +if (codes[0]?.kind === "url") console.log(codes[0].text) +``` + ### Errors `raiseHand` throws only before the handoff URL exists — while nobody has been @@ -369,8 +422,10 @@ handraise brings the same handoff to Solari browsers, which have no native live view (Solari's VNC is desktop-only), as a portable library instead — less polished, and it works where those don't. What the hosted live views do not have is the second mode: an approval is a yes-or-no on one screenshot, no -live session exposed at all, answerable from a chat channel. Its scope stops -at the handoff, not wall detection +live session exposed at all, answerable from a chat channel. Nor do they have +an answer to a device-change check — a QR code a phone is asked to scan, on +the phone's own screen — which handraise reads off the page and hands over as +a link. Its scope stops at the handoff, not wall detection ([`docs/adr/0005`](docs/adr/0005-handoff-not-wall-detection.md)). ## Security @@ -444,14 +499,17 @@ a 2FA, and each handoff consumes one sandbox, destroyed when it ends. ## Verified how -Benchmark method and raw data: [`benchmarks/`](benchmarks/README.md). The four +Benchmark method and raw data: [`benchmarks/`](benchmarks/README.md). The five platform measurements the design rests on — transport, screencast, input -injection, session lifetime — are in +injection, session lifetime, QR decoding — are in [`docs/measurements/`](docs/measurements/README.md). The e2e test drives the whole loop with no mocks: a Solari browser signs into a TOTP-protected demo app ([`test-app/`](test-app/), deployed into a sandbox), hits the 2FA wall, raises its hand, a scripted "human" types the code through the real handoff UI, and -the test asserts the signed-in page — ~6s end to end. Injected events arrive +the test asserts the signed-in page — ~6s end to end. The same run then drives +the QR passthrough: the app shows a device-change code, the human asks for a +scan, and the link that comes back is fetched from outside the browser to reach +the confirmation page. Injected events arrive with `isTrusted: true`. ## Limitations (v1) @@ -464,11 +522,12 @@ with `isTrusted: true`. - An approval shows the page as it was when the agent asked. If the page changes underneath (a session expiring, a redirect), the human is deciding on a stale picture — the frame is not refreshed. -- A verification that shows a QR code to scan (reCAPTCHA's "scan to verify - you're human") needs a second screen today: open the handoff link on a - laptop and scan it with the phone — the phone cannot scan its own display. - Decoding the QR from the live frame and handing the phone the link is - planned. +- The QR passthrough is **untested against reCAPTCHA itself**: its demo never + served the scan-to-verify variant, which Google shows at its own discretion. + The mechanism is proven end to end against a page that behaves the same way + ([measurement 05 §7](docs/measurements/05-qr.md)). A code drawn below ~120 + CSS pixels does not decode, and three or more codes on one screen are not + attempted. - TypeScript/Node only for now. ## Contributing diff --git a/bun.lock b/bun.lock index 7e6d36e..0081651 100644 --- a/bun.lock +++ b/bun.lock @@ -7,6 +7,7 @@ "dependencies": { "@solarisdk/browser": "^0.1.2", "@solarisdk/sdk": "^0.1.2", + "jsqr": "1.4.0", "qrcode-terminal": "^0.12.0", "ws": "^8.21.3", }, @@ -15,12 +16,14 @@ "@ai-sdk/openai-compatible": "^3.0.41", "@biomejs/biome": "^2.0.0", "@oxlint/plugins": "1.80.0", + "@types/qrcode": "^1.5.6", "@types/qrcode-terminal": "^0.12.2", "@types/ws": "^8.5.13", "ai": "^7.0.87", "bun-types": "^1.4.0", "oxlint": "1.80.0", "playwright-core": "^1.62.1", + "qrcode": "^1.5.4", "tsup": "^8.5.1", "typescript": "^5.7.2", }, @@ -226,6 +229,8 @@ "@types/node": ["@types/node@26.4.0", "", { "dependencies": { "undici-types": "~8.3.0" } }, "sha512-faiGnoIrLH/V8cibOMEAZ8pMw6oXqSukl29ra4mN8GdaB2ZewzeaLj+INpV5N+Z1eKWzY+IzaIZH2EIR6YZRNQ=="], + "@types/qrcode": ["@types/qrcode@1.5.6", "", { "dependencies": { "@types/node": "*" } }, "sha512-te7NQcV2BOvdj2b1hCAHzAoMNuj65kNBMz0KBaxM6c3VGBOhU0dURQKOtH8CFNI/dsKkwlv32p26qYQTWoB5bw=="], + "@types/qrcode-terminal": ["@types/qrcode-terminal@0.12.2", "", {}, "sha512-v+RcIEJ+Uhd6ygSQ0u5YYY7ZM+la7GgPbs0V/7l/kFs2uO4S8BcIUEMoP7za4DNIqNnUD5npf0A/7kBhrCKG5Q=="], "@types/ws": ["@types/ws@8.18.1", "", { "dependencies": { "@types/node": "*" } }, "sha512-ThVF6DCVhA8kUGy+aazFQ4kXQ7E1Ty7A3ypFOe0IcJV8O/M511G99AW24irKrW56Wt44yG9+ij8FaqoBGkuBXg=="], @@ -238,6 +243,10 @@ "ai": ["ai@7.0.87", "", { "dependencies": { "@ai-sdk/gateway": "4.0.70", "@ai-sdk/provider": "4.0.9", "@ai-sdk/provider-utils": "5.0.34" }, "peerDependencies": { "zod": "^3.25.76 || ^4.1.8" } }, "sha512-/hrT7toRx8vLIyr/lTKOOPDxCGdxk2tVs5viHDwIPlUsge5FBaLe9h3BhPAr7cOmEAXkN/Cf2+2N8HYGCjJbHg=="], + "ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], + + "ansi-styles": ["ansi-styles@4.3.0", "", { "dependencies": { "color-convert": "^2.0.1" } }, "sha512-zbB9rCJAT1rbjiVDb2hqKFHNYLxgtk8NURxZ3IZwD3F6NtxbXZQCnnSi1Lkx+IDohdPlFp222wVALIheZJQSEg=="], + "any-promise": ["any-promise@1.3.0", "", {}, "sha512-7UvmKalWRt1wgjL1RrGxoSJW/0QZFIegpeGvZG9kjp8vrRu55XTHbwnqq2GpXm9uLbcuhxm3IqX9OB4MZR1b2A=="], "bun-types": ["bun-types@1.4.0", "", { "dependencies": { "@types/node": "*" } }, "sha512-iIKw23BspnQQYd3prITOBxeUsxBHnwzX6YJfGMuNOZzeNcMmVqzIIVGRm1l69ogaPQmb4wB6BN8mA5bE9YuC5Q=="], @@ -246,8 +255,16 @@ "cac": ["cac@6.7.14", "", {}, "sha512-b6Ilus+c3RrdDk+JhLKUAQfzzgLEPy6wcXqS7f/xe1EETvsDP6GORG7SFuOs6cID5YkqchW/LXZbX5bc8j7ZcQ=="], + "camelcase": ["camelcase@5.3.1", "", {}, "sha512-L28STB170nwWS63UjtlEOE3dldQApaJXZkOI1uMFfzf3rRuPegHaHesyee+YxQ+W6SvRDQV6UrdOdRiR153wJg=="], + "chokidar": ["chokidar@4.0.3", "", { "dependencies": { "readdirp": "^4.0.1" } }, "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA=="], + "cliui": ["cliui@6.0.0", "", { "dependencies": { "string-width": "^4.2.0", "strip-ansi": "^6.0.0", "wrap-ansi": "^6.2.0" } }, "sha512-t6wbgtoCXvAzst7QgXxJYqPt0usEfbgQdftEPbLL/cvv6HPE5VgvqCuAIDR0NgU52ds6rFwqrgakNLrHEjCbrQ=="], + + "color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], + + "color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], + "commander": ["commander@4.1.1", "", {}, "sha512-NOKm8xhkzAjzFx8B2v5OAHT+u5pRQc2UCa2Vq9jYL/31o2wi9mxBA7LIFs3sV5VSC49z6pEhfbMULvShKj26WA=="], "confbox": ["confbox@0.1.8", "", {}, "sha512-RMtmw0iFkeR4YV+fUOSucriAQNb9g8zFR52MWCtl+cCZOFRNL6zeB395vPzFhEjjn4fMxXudmELnl/KF/WrK6w=="], @@ -256,26 +273,42 @@ "debug": ["debug@4.4.3", "", { "dependencies": { "ms": "^2.1.3" } }, "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA=="], + "decamelize": ["decamelize@1.2.0", "", {}, "sha512-z2S+W9X73hAUUki+N+9Za2lBlun89zigOyGrsax+KUQ6wKW4ZoWpEYBkGhQjwAjjDCkWxhY0VKEhk8wzY7F5cA=="], + + "dijkstrajs": ["dijkstrajs@1.0.3", "", {}, "sha512-qiSlmBq9+BCdCA/L46dw8Uy93mloxsPSbwnm5yrKn2vMPiy8KyAskTF6zuV/j5BMsmOGZDPs7KjU+mjb670kfA=="], + + "emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], + "esbuild": ["esbuild@0.27.7", "", { "optionalDependencies": { "@esbuild/aix-ppc64": "0.27.7", "@esbuild/android-arm": "0.27.7", "@esbuild/android-arm64": "0.27.7", "@esbuild/android-x64": "0.27.7", "@esbuild/darwin-arm64": "0.27.7", "@esbuild/darwin-x64": "0.27.7", "@esbuild/freebsd-arm64": "0.27.7", "@esbuild/freebsd-x64": "0.27.7", "@esbuild/linux-arm": "0.27.7", "@esbuild/linux-arm64": "0.27.7", "@esbuild/linux-ia32": "0.27.7", "@esbuild/linux-loong64": "0.27.7", "@esbuild/linux-mips64el": "0.27.7", "@esbuild/linux-ppc64": "0.27.7", "@esbuild/linux-riscv64": "0.27.7", "@esbuild/linux-s390x": "0.27.7", "@esbuild/linux-x64": "0.27.7", "@esbuild/netbsd-arm64": "0.27.7", "@esbuild/netbsd-x64": "0.27.7", "@esbuild/openbsd-arm64": "0.27.7", "@esbuild/openbsd-x64": "0.27.7", "@esbuild/openharmony-arm64": "0.27.7", "@esbuild/sunos-x64": "0.27.7", "@esbuild/win32-arm64": "0.27.7", "@esbuild/win32-ia32": "0.27.7", "@esbuild/win32-x64": "0.27.7" }, "bin": { "esbuild": "bin/esbuild" } }, "sha512-IxpibTjyVnmrIQo5aqNpCgoACA/dTKLTlhMHihVHhdkxKyPO1uBBthumT0rdHmcsk9uMonIWS0m4FljWzILh3w=="], "eventsource-parser": ["eventsource-parser@3.1.1", "", {}, "sha512-EKN1vKAMcZ8MlYMpaNuxN6R9yakzH6uajHcHVTqWJzvu5pWw9DyhbP35HH8MVBQ+dZjAfDxk+A8NiR9KWaXiyQ=="], "fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="], + "find-up": ["find-up@4.1.0", "", { "dependencies": { "locate-path": "^5.0.0", "path-exists": "^4.0.0" } }, "sha512-PpOwAdQ/YlXQ2vj8a3h8IipDuYRi3wceVQQGYWxNINccq40Anw7BlsEXCMbt1Zt+OLA6Fq9suIpIWD0OsnISlw=="], + "fix-dts-default-cjs-exports": ["fix-dts-default-cjs-exports@1.0.1", "", { "dependencies": { "magic-string": "^0.30.17", "mlly": "^1.7.4", "rollup": "^4.34.8" } }, "sha512-pVIECanWFC61Hzl2+oOCtoJ3F17kglZC/6N94eRWycFgBH35hHx0Li604ZIzhseh97mf2p0cv7vVrOZGoqhlEg=="], "fsevents": ["fsevents@2.3.3", "", { "os": "darwin" }, "sha512-5xoDfX+fL7faATnagmWPpbFtwh/R77WmMMqqHGS65C3vvB0YHrgF+B1YmZ3441tMj5n63k0212XNoJwzlhffQw=="], + "get-caller-file": ["get-caller-file@2.0.5", "", {}, "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg=="], + + "is-fullwidth-code-point": ["is-fullwidth-code-point@3.0.0", "", {}, "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg=="], + "joycon": ["joycon@3.1.1", "", {}, "sha512-34wB/Y7MW7bzjKRjUKTa46I2Z7eV62Rkhva+KkopW7Qvv/OSWBqvkSY7vusOPrNuZcUG3tApvdVgNB8POj3SPw=="], "json-schema": ["json-schema@0.4.0", "", {}, "sha512-es94M3nTIfsEPisRafak+HDLfHXnKBhV3vU5eqPcS3flIWqcxJWgXHXiey3YrpaNsanY5ei1VoYEbOzijuq9BA=="], + "jsqr": ["jsqr@1.4.0", "", {}, "sha512-dxLob7q65Xg2DvstYkRpkYtmKm2sPJ9oFhrhmudT1dZvNFFTlroai3AWSpLey/w5vMcLBXRgOJsbXpdN9HzU/A=="], + "lilconfig": ["lilconfig@3.1.3", "", {}, "sha512-/vlFKAoH5Cgt3Ie+JLhRbwOsCQePABiU3tJ1egGvyQ+33R/vcwM2Zl2QR/LzjsBeItPt3oSVXapn+m4nQDvpzw=="], "lines-and-columns": ["lines-and-columns@1.2.4", "", {}, "sha512-7ylylesZQ/PV29jhEDl3Ufjo6ZX7gCqJr5F7PKrqc93v7fzSymt1BpwEU8nAUXs8qzzvqhbjhK5QZg6Mt/HkBg=="], "load-tsconfig": ["load-tsconfig@0.2.5", "", {}, "sha512-IXO6OCs9yg8tMKzfPZ1YmheJbZCiEsnBdcB03l0OcfK9prKnJb96siuHCr5Fl37/yo9DnKU+TLpxzTUspw9shg=="], + "locate-path": ["locate-path@5.0.0", "", { "dependencies": { "p-locate": "^4.1.0" } }, "sha512-t7hw9pI+WvuwNJXwk5zVHpyhIqzg2qTlklJOf0mVxGSbe3Fp2VieZcduNYjaLDoy6p9uGpQEGWG87WpMKlNq8g=="], + "magic-string": ["magic-string@0.30.21", "", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.5" } }, "sha512-vd2F4YUyEXKGcLHoq+TEyCjxueSeHnFxyyjNp80yg0XV4vUhnDer/lvvlqM/arB5bXQN5K2/3oinyCRyx8T2CQ=="], "mlly": ["mlly@1.8.2", "", { "dependencies": { "acorn": "^8.16.0", "pathe": "^2.0.3", "pkg-types": "^1.3.1", "ufo": "^1.6.3" } }, "sha512-d+ObxMQFmbt10sretNDytwt85VrbkhhUA/JBGm1MPaWJ65Cl4wOgLaB1NYvJSZ0Ef03MMEU/0xpPMXUIQ29UfA=="], @@ -288,8 +321,16 @@ "oxlint": ["oxlint@1.80.0", "", { "optionalDependencies": { "@oxlint/binding-android-arm-eabi": "1.80.0", "@oxlint/binding-android-arm64": "1.80.0", "@oxlint/binding-darwin-arm64": "1.80.0", "@oxlint/binding-darwin-x64": "1.80.0", "@oxlint/binding-freebsd-x64": "1.80.0", "@oxlint/binding-linux-arm-gnueabihf": "1.80.0", "@oxlint/binding-linux-arm-musleabihf": "1.80.0", "@oxlint/binding-linux-arm64-gnu": "1.80.0", "@oxlint/binding-linux-arm64-musl": "1.80.0", "@oxlint/binding-linux-ppc64-gnu": "1.80.0", "@oxlint/binding-linux-riscv64-gnu": "1.80.0", "@oxlint/binding-linux-riscv64-musl": "1.80.0", "@oxlint/binding-linux-s390x-gnu": "1.80.0", "@oxlint/binding-linux-x64-gnu": "1.80.0", "@oxlint/binding-linux-x64-musl": "1.80.0", "@oxlint/binding-openharmony-arm64": "1.80.0", "@oxlint/binding-win32-arm64-msvc": "1.80.0", "@oxlint/binding-win32-ia32-msvc": "1.80.0", "@oxlint/binding-win32-x64-msvc": "1.80.0" }, "peerDependencies": { "oxlint-tsgolint": ">=7.0.2001", "vite-plus": "*" }, "optionalPeers": ["oxlint-tsgolint", "vite-plus"], "bin": { "oxlint": "bin/oxlint" } }, "sha512-5nTiSps4qdbCWLbxzuO00alHkEO2exR9YMN/ig6QXWrLsYSG0KaObOAM+l6oU2LcKPWoSAGYbkZIGEu1ViiWKA=="], + "p-limit": ["p-limit@2.3.0", "", { "dependencies": { "p-try": "^2.0.0" } }, "sha512-//88mFWSJx8lxCzwdAABTJL2MyWB12+eIY7MDL2SqLmAkeKU9qxRvWuSyTjm3FUmpBEMuFfckAIqEaVGUDxb6w=="], + + "p-locate": ["p-locate@4.1.0", "", { "dependencies": { "p-limit": "^2.2.0" } }, "sha512-R79ZZ/0wAxKGu3oYMlz8jy/kbhsNrS7SKZ7PxEHBgJ5+F2mtFW2fK2cOtBh1cHYkQsbzFV7I+EoRKe6Yt0oK7A=="], + + "p-try": ["p-try@2.2.0", "", {}, "sha512-R4nPAVTAU0B9D35/Gk3uJf/7XYbQcyohSKdvAxIRSNghFl4e71hVoGnBNQz9cWaXxO2I10KTC+3jMdvvoKw6dQ=="], + "patchright-core": ["patchright-core@1.62.2", "", { "bin": { "patchright-core": "cli.js" } }, "sha512-XDl3HB/Kjm9ZBFn4ygD5mxdsqK4aIWNGXdea46XAGd0hvHuJJ84w1M83JqofpXIbD2aJfQ2Rtz4YcqPVrxjSzw=="], + "path-exists": ["path-exists@4.0.0", "", {}, "sha512-ak9Qy5Q7jYb2Wwcey5Fpvg2KoAc/ZIhLSLOSBmRmygPsGwkVVt0fZa0qrtMz+m6tJTAHfZQ8FnmB4MG4LWy7/w=="], + "pathe": ["pathe@2.0.3", "", {}, "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w=="], "picocolors": ["picocolors@1.1.1", "", {}, "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="], @@ -302,18 +343,32 @@ "playwright-core": ["playwright-core@1.62.1", "", { "bin": { "playwright-core": "cli.js" } }, "sha512-wPYSwEBJY9GHraISXqyqtx0na0LpO3XEX7jNDhntbex7tzUS7kLnZsOlFruFJB4Hi/rhDMjXGqHewDZ68nYZVw=="], + "pngjs": ["pngjs@5.0.0", "", {}, "sha512-40QW5YalBNfQo5yRYmiw7Yz6TKKVr3h6970B2YE+3fQpsWcrbj1PzJgxeJ19DRQjhMbKPIuMY8rFaXc8moolVw=="], + "postcss-load-config": ["postcss-load-config@6.0.1", "", { "dependencies": { "lilconfig": "^3.1.1" }, "peerDependencies": { "jiti": ">=1.21.0", "postcss": ">=8.0.9", "tsx": "^4.8.1", "yaml": "^2.4.2" }, "optionalPeers": ["jiti", "postcss", "tsx", "yaml"] }, "sha512-oPtTM4oerL+UXmx+93ytZVN82RrlY/wPUV8IeDxFrzIjXOLF1pN+EmKPLbubvKHT2HC20xXsCAH2Z+CKV6Oz/g=="], + "qrcode": ["qrcode@1.5.4", "", { "dependencies": { "dijkstrajs": "^1.0.1", "pngjs": "^5.0.0", "yargs": "^15.3.1" }, "bin": { "qrcode": "bin/qrcode" } }, "sha512-1ca71Zgiu6ORjHqFBDpnSMTR2ReToX4l1Au1VFLyVeBTFavzQnv5JxMFr3ukHVKpSrSA2MCk0lNJSykjUfz7Zg=="], + "qrcode-terminal": ["qrcode-terminal@0.12.0", "", { "bin": { "qrcode-terminal": "./bin/qrcode-terminal.js" } }, "sha512-EXtzRZmC+YGmGlDFbXKxQiMZNwCLEO6BANKXG4iCtSIM0yqc/pappSx3RIKr4r0uh5JsBckOXeKrB3Iz7mdQpQ=="], "readdirp": ["readdirp@4.1.2", "", {}, "sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg=="], + "require-directory": ["require-directory@2.1.1", "", {}, "sha512-fGxEI7+wsG9xrvdjsrlmL22OMTTiHRwAMroiEeMgq8gzoLC/PQr7RsRDSTLUg/bZAZtF+TVIkHc6/4RIKrui+Q=="], + + "require-main-filename": ["require-main-filename@2.0.0", "", {}, "sha512-NKN5kMDylKuldxYLSUfrbo5Tuzh4hd+2E8NPPX02mZtn1VuREQToYe/ZdlJy+J3uCpfaiGF05e7B8W0iXbQHmg=="], + "resolve-from": ["resolve-from@5.0.0", "", {}, "sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw=="], "rollup": ["rollup@4.63.1", "", { "dependencies": { "@types/estree": "1.0.9" }, "optionalDependencies": { "@napi-rs/lzma-linux-x64-gnu": "1.5.1", "@rollup/rollup-android-arm-eabi": "4.63.1", "@rollup/rollup-android-arm64": "4.63.1", "@rollup/rollup-darwin-arm64": "4.63.1", "@rollup/rollup-darwin-x64": "4.63.1", "@rollup/rollup-freebsd-arm64": "4.63.1", "@rollup/rollup-freebsd-x64": "4.63.1", "@rollup/rollup-linux-arm-gnueabihf": "4.63.1", "@rollup/rollup-linux-arm-musleabihf": "4.63.1", "@rollup/rollup-linux-arm64-gnu": "4.63.1", "@rollup/rollup-linux-arm64-musl": "4.63.1", "@rollup/rollup-linux-loong64-gnu": "4.63.1", "@rollup/rollup-linux-loong64-musl": "4.63.1", "@rollup/rollup-linux-ppc64-gnu": "4.63.1", "@rollup/rollup-linux-ppc64-musl": "4.63.1", "@rollup/rollup-linux-riscv64-gnu": "4.63.1", "@rollup/rollup-linux-riscv64-musl": "4.63.1", "@rollup/rollup-linux-s390x-gnu": "4.63.1", "@rollup/rollup-linux-x64-gnu": "4.63.1", "@rollup/rollup-linux-x64-musl": "4.63.1", "@rollup/rollup-openbsd-x64": "4.63.1", "@rollup/rollup-openharmony-arm64": "4.63.1", "@rollup/rollup-win32-arm64-msvc": "4.63.1", "@rollup/rollup-win32-ia32-msvc": "4.63.1", "@rollup/rollup-win32-x64-gnu": "4.63.1", "@rollup/rollup-win32-x64-msvc": "4.63.1", "fsevents": "~2.3.2" }, "bin": { "rollup": "dist/bin/rollup" } }, "sha512-3Df9jsstwhccuEfmAMi9l8XUh/GOkVObmFTU7CCVBysEbcOZLl84jCtaAZMcPiMz2EGKsATzQcU+Xr3n/wU6cg=="], + "set-blocking": ["set-blocking@2.0.0", "", {}, "sha512-KiKBS8AnWGEyLzofFfmvKwpdPzqiy16LvQfK3yv/fVH7Bj13/wl3JSR1J+rfgRE9q7xUJK4qvgS8raSOeLUehw=="], + "source-map": ["source-map@0.7.6", "", {}, "sha512-i5uvt8C3ikiWeNZSVZNWcfZPItFQOsYTUAOkcUPGd8DqDy1uOUikjt5dG+uRlwyvR108Fb9DOd4GvXfT0N2/uQ=="], + "string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], + + "strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], + "sucrase": ["sucrase@3.35.1", "", { "dependencies": { "@jridgewell/gen-mapping": "^0.3.2", "commander": "^4.0.0", "lines-and-columns": "^1.1.6", "mz": "^2.7.0", "pirates": "^4.0.1", "tinyglobby": "^0.2.11", "ts-interface-checker": "^0.1.9" }, "bin": { "sucrase": "bin/sucrase", "sucrase-node": "bin/sucrase-node" } }, "sha512-DhuTmvZWux4H1UOnWMB3sk0sbaCVOoQZjv8u1rDoTV0HTdGem9hkAZtl4JZy8P2z4Bg0nT+YMeOFyVr4zcG5Tw=="], "thenify": ["thenify@3.3.1", "", { "dependencies": { "any-promise": "^1.0.0" } }, "sha512-RVZSIV5IG10Hk3enotrhvz0T9em6cyHBLkH/YAZuKqd8hRkKhSfCGIcP2KUY0EPxndzANBmNllzWPwak+bheSw=="], @@ -338,8 +393,18 @@ "undici-types": ["undici-types@8.3.0", "", {}, "sha512-j375ScV60dom+YkPFIfTLcOiPxkN/buHz5GobjLhixFuANaNs3C9l4GmrWqejgXWJ7BbJcFYpTEUkS1Ge8bpZQ=="], + "which-module": ["which-module@2.0.1", "", {}, "sha512-iBdZ57RDvnOR9AGBhML2vFZf7h8vmBjhoaZqODJBFWHVtKkDmKuHai3cx5PgVMrX5YDNp27AofYbAwctSS+vhQ=="], + + "wrap-ansi": ["wrap-ansi@6.2.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-r6lPcBGxZXlIcymEu7InxDMhdW0KDxpLgoFLcguasxCaJ/SOIZwINatK9KY/tf+ZrlywOKU0UDj3ATXUBfxJXA=="], + "ws": ["ws@8.21.3", "", { "peerDependencies": { "bufferutil": "^4.0.1", "utf-8-validate": ">=5.0.2" }, "optionalPeers": ["bufferutil", "utf-8-validate"] }, "sha512-201TZ/kPWxoPr/OKWjquZR1SWKXcvxdH+e1xrx89b3YbmzLMFCLfnaG1HFIgWzJOEWZ7MvpK++odZufgYR50Rw=="], + "y18n": ["y18n@4.0.3", "", {}, "sha512-JKhqTOwSrqNA1NY5lSztJ1GrBiUodLMmIZuLiDaMRJ+itFd+ABVE8XBjOvIWL+rSqNDC74LCSFmlb/U4UZ4hJQ=="], + + "yargs": ["yargs@15.4.1", "", { "dependencies": { "cliui": "^6.0.0", "decamelize": "^1.2.0", "find-up": "^4.1.0", "get-caller-file": "^2.0.1", "require-directory": "^2.1.1", "require-main-filename": "^2.0.0", "set-blocking": "^2.0.0", "string-width": "^4.2.0", "which-module": "^2.0.0", "y18n": "^4.0.0", "yargs-parser": "^18.1.2" } }, "sha512-aePbxDmcYW++PaqBsJ+HYUFwCdv4LVvdnhBy78E57PIor8/OVvhMrADFFEDh8DHDFRv/O9i3lPhsENjO7QX0+A=="], + + "yargs-parser": ["yargs-parser@18.1.3", "", { "dependencies": { "camelcase": "^5.0.0", "decamelize": "^1.2.0" } }, "sha512-o50j0JeToy/4K6OZcaQmW6lyXXKhq7csREXcDwk2omFPJEwUNOVtJKvmDr9EI1fAJZUyZcRF7kxGBWmRXudrCQ=="], + "zod": ["zod@4.5.4", "", {}, "sha512-sC95tT5iHHH9gtpj6A81kh+NEaRAUFN+qlUPDUbRfOMvNf5QCBqsb3WgvnpVtK5Y+4UfA6KqufotuTvMGiTlsA=="], } } diff --git a/docs/adr/0008-qr-passthrough.md b/docs/adr/0008-qr-passthrough.md new file mode 100644 index 0000000..cbdcc33 --- /dev/null +++ b/docs/adr/0008-qr-passthrough.md @@ -0,0 +1,212 @@ +# 0008 — QR passthrough: the agent reads the code, the phone gets the link + +- **Status:** accepted +- **Date:** 2026-09-02 + +## Context + +A growing class of walls asks for a *second device*. reCAPTCHA's scan-to-verify +variant, a WhatsApp Web login, an authenticator enrolment, a payment code: the +site draws a QR code and says "scan this with your phone". + +handraise's whole answer to a wall is to put a human on a phone in front of the +page. That answer breaks here, and it breaks in a way that is funny once and +then expensive: the human is holding the phone the site is asking for, and the +code is on that phone's screen. **A phone cannot scan itself.** Before this, +the only way through was a second device — which defeats the point of a handoff +that was supposed to take twenty seconds. + +The information is right there. The code is a string, usually a URL, and the +agent has the page it is drawn on. + +## Decision + +**A human-initiated scan.** The phone gains one control in the key bar, +`Scan QR`. It sends `{ "type": "scanqr" }`; the agent takes a fresh +full-resolution `page.screenshot({ type: "png" })`, decodes it, and answers +`{ "type": "links", "links": [...], "source": "qr" }`. The phone shows a sheet +with what each code said, an **Open in new tab** button for the schemes below, +and **Copy** for everything. + +Five decisions inside that, each one measured or argued rather than assumed +([Measurement 05](../measurements/05-qr.md)). + +**Decode in the agent process, not in the page.** `BarcodeDetector` does not +exist in Solari's Chromium (measured, §1), so "let the browser do it" was not +available. It would have been the wrong shape anyway: running the decode via +`page.evaluate` puts handraise's code in the realm of whatever site the agent +got stuck on, where the page can replace `BarcodeDetector` and answer with a +link of its choosing — a link a human is then invited to open on their phone. +The decode is `jsqr` plus a PNG decoder written against `node:zlib` +(`src/core/png.ts`, ~180 lines): one new pure-JavaScript dependency, no native +build on any platform an agent runs on. + +**A fresh screenshot, not the cast frame.** The phone is already looking at a +picture of the page, and reusing it would cost nothing. It is not good enough: +the cast is 800 px wide at JPEG quality 60, chosen for a form field, and it +fails on symbols the screenshot reads (measured, §3 — 180 px and 120 px fail +from the frame and decode from the screenshot, with a luck-dependent band in +between). The screenshot costs 239 ms p50, which is the price of the difference +between "works" and "works when the site draws its code large". + +**On request, and rate-limited.** No auto-scan of every frame: a scan is a +screenshot plus a decode on a stream that already paces itself to a phone's +link, and 99% of pages have no code on them. One scan per 2 s, enforced in the +core rather than on the phone — the handoff URL is a bearer credential and the +socket behind it is reachable from any HTTP client, so the phone's own floor is +a courtesy, not a limit. + +**An allowlist of openable schemes, checked twice.** `http:`, `https:`, `tel:`, +`mailto:`, `otpauth:` may be opened; everything else is shown as text with a +Copy button and no anchor. An allowlist rather than a blocklist because the +interesting half is the half nobody thinks of: `javascript:` and `data:` are +the two everyone remembers, and `intent:`, `file:`, `content:` and whatever a +phone browser ships next year are the ones a blocklist would have let through. +The agent classifies, and the phone checks the scheme again before it builds +the anchor — the `kind` field crosses a socket a stranger holding the link can +write to, so the page must not have to trust it. The anchor carries +`rel="noopener noreferrer"`: the opened site gets no handle on the tab holding +a live handoff, and is not told the handoff URL it came from. + +**The agent never opens anything.** It reads and classifies; the fetching is +the human's, in their own browser. An agent process that followed a URL out of +a hostile page would be an SSRF primitive with the agent's own network position. + +**And it never blocks on the decode.** The PNG decode plus up to three `jsQR` +passes is pure synchronous CPU: on a 3840x2160 screenshot it held the event +loop for **2132 ms, during which a 5 ms heartbeat did not tick once** +(measured, §6). That loop is the frame pump, the human's handback, the timeout +and the browser's disconnect. So the work runs in a worker thread — +`dist/qr-worker.js`, started at the first scan of a handoff, terminated when it +settles — and the same decode costs the loop one millisecond. The worker also +buys the only lever there is over a decode that will not finish: a six-second +deadline and `terminate()`. + +**Every size is bounded before anything is allocated.** A PNG's header is the +only part of it that is cheap to believe, and everything downstream is sized +from it, so it is checked first: 8192 pixels a side, 24 megapixels in total, 32 +MB of compressed image data, and the exact inflated length the header implies +passed to `inflateSync` as `maxOutputLength` and then required to match. The 2x +retry allocates four times the source, so it is refused above 40 megapixels +rather than taking a half-gigabyte step for a code it probably cannot read. + +Takeover only. An approval is one screenshot of a moment, not a live page — +there is nothing to scan and nothing a scan could change. + +## Alternatives + +**Auto-scan every cast frame.** Rejected on cost and on false positives: a +decode per frame on a stream that runs at ~14 fps, to answer a question almost +every page answers "no" to, and a sheet that opens itself while the human is +typing. + +**Decode the cast frame on the phone.** Attractive — no protocol change, no +agent work, and the phone already has `BarcodeDetector` on iOS. Rejected on the +same measurement that killed reusing the frame in the agent (§3): the picture +the phone holds is the one that does not decode. It would also have shipped a +feature whose reliability depended on which phone the human happened to hold. + +**Send a screenshot to the phone and let it decode.** The phone would need the +full-resolution PNG — 43.7 KB here, more on a real page — over a link that is +already pacing a live cast, to run a decode that costs 54 ms in Node. All of +the bandwidth, none of the control, and still phone-dependent. + +**Let the agent open the link itself.** It is the obvious shortcut and it is +wrong twice: the site is asking for a *different device* on purpose, so opening +it from the browser that showed the code defeats the check it is making; and it +turns any page the agent lands on into a request the agent will make. + +**`pngjs` instead of a hand-written PNG decoder.** A second dependency to read +four colour types at one bit depth. The decoder is ~180 lines against +`node:zlib`, refuses everything it has not been fed, and is tested against a +real cloud-browser screenshot. + +## Consequences + +- One new runtime dependency: `jsqr` (pure JavaScript, no dependencies of its + own). `qrcode` is added as a **dev** dependency, for the test app's page and + for generating test images. +- The protocol grows one message in each direction (`scanqr`, `links`). The + wide event grows `qrScans` and `qrHits`; `qrScans - qrHits` is the number + worth watching, because it is either a page with no code or a decode that + failed. +- `scanQrLinks(png)` and `OPENABLE_SCHEMES` are exported: an agent that wants + to read a code without a human can, and the dist smoke uses it to prove the + CommonJS interop survives bundling — the failure mode that once broke + `qrcode-terminal` in `dist` while every bun test stayed green. +- **A symbol below ~120 CSS px will not decode** (§3). The sheet says nothing + was found; the human can zoom or scroll the remote page and scan again. +- **One decode is not enough.** A code the page drew at a resampled size can be + large, centred and sharp and still not be *located*, because `jsQR` + thresholds in fixed 8x8 blocks and a 4.9-pixel module grid straddles them + (§5). `scanImage` therefore looks up to three times — as it came, at 2x, then + four overlapping corners — and only when the previous look found nothing. A + page with no code pays all three, about 320 ms of CPU. This was found by the + live e2e after every offline test had passed, which is the argument for + having one. +- **Two codes on one screen** need the tiled second pass to be found at all + (§4). Three or more are not attempted: `MAX_CODES` is 2. +- **reCAPTCHA itself is untested.** Its demo never served the QR variant + (§7). The mechanism is proven end to end against the test app's `/qr` page in + the live e2e, and the README says as much rather than implying more. +- A hostile page can put any string in a QR code and have a human read it on a + phone. That is the residual risk, and it is bounded by the allowlist, by the + page applying the whole rule again itself, by `noreferrer`, and by the fact + that opening it is an explicit act by a person who can see the whole link — + which is why the sheet never truncates it. + + **"The whole link" has to mean the link that opens.** A raw payload beside an + anchor that resolves it is two strings, and the difference is invisible: + `https://аpple.com` with a Cyrillic а reads as apple.com and lands on + `xn--pple-43d.com`, and a U+202E override reverses the visible tail of a + path. So an openable link is shown as `new URL(text).href` — punycode host, + percent-encoded overrides — and the card says so when that differs from the + payload. Copy takes the same string. Nothing is truncated, so the argument + above still holds; it just stopped being false. + +- **`tel:` and `otpauth:` are not openable, and that is a reversal.** They were + on the allowlist because a device-change flow genuinely uses them. They are + off it because both are *actions* rather than pages, and both are one tap: + `tel:*21*1234567890%23` is a call-forwarding sequence handed to a dialler, + and an `otpauth:` URI enrols an attacker-chosen TOTP secret in the human's + authenticator — the feature working exactly as designed, and exactly what a + phisher wants from it. Neither is worth a tap taken from a page nobody + vetted. They are still decoded, still shown in full and still copyable, under + a label that says what they are ("Phone number", "Authenticator secret") so + the human hands them to the right app deliberately. Only `http`, `https` and + `mailto` keep an Open button. + +- **The decode no longer blocks the event loop, and `scanQrLinks` still does.** + The exported function is synchronous by design — a caller's own thread is + their business — and the handoff path uses `createQrScanner()`, which is the + same decode on a worker. Both are exported; the ADR's argument only covers + the second. +- **The relay's backpressure pause delays, it does not drop.** When the agent's + socket is full the relay stops reading the human's, having first written the + message that triggered it. Anything already behind that in the human's socket + buffer — a handback among them — waits for the agent to drain, or for it to + go away, at which point `closePeer` resumes the human. Bounded by + drain-or-close and never lost, but not instant. Separating terminal messages + out would mean parsing before the flow-control decision, which is exactly the + work the 4 KiB pre-parse cap exists to avoid. + +- **A worker is a file, not a function.** `dist/qr-worker.js` is resolved at + runtime with `new URL("./qr-worker.js", import.meta.url)`, so a consumer who + re-bundles handraise has to keep it next to the entry it resolves from. The + dist smoke runs the worker under node against the shipped artifact, because + this is the one part of the package a bundler can silently drop. + +- **`jsqr` is a bet, not just a dependency**, and it is pinned to exactly + `1.4.0` rather than a range. That release is from January 2021 and + `cozmo/jsQR` is archived. The bet is taken knowingly: zero transitive + dependencies, Apache-2.0, pure JavaScript with no `eval`, so a bad decode is + a crash or a stall and never an execution primitive — and it now runs in a + worker, where a stall is a terminated thread. The alternatives in the + ecosystem are WASM or native, which trades an archived dependency for a build + step on every platform an agent runs on. + + **What would trigger a fork or a replacement:** a decode that is wrong rather + than absent (a payload that is not what the symbol encodes), a crash that the + worker deadline cannot contain, or a CVE. The committed fixtures and the dist + smoke are the corpus a fork would have to keep passing, and `scanImage` is + the only seam it would have to fit. diff --git a/docs/adr/README.md b/docs/adr/README.md index ea076d9..a7e0fd3 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -15,6 +15,7 @@ pre-publish security review — they document the history, they do not invent it | [0005](0005-handoff-not-wall-detection.md) | handraise is the handoff mechanism, not the wall detection | accepted | Scope decision | | [0006](0006-approval-mode.md) | Approval mode: one screenshot, a hold on yes | accepted | Scope decision | | [0007](0007-channels.md) | Channels: an in-process hook, not a second WebSocket client | accepted | Scope decision | +| [0008](0008-qr-passthrough.md) | QR passthrough: the agent reads the code, the phone gets the link | accepted | [Measurement 05](../measurements/05-qr.md) | ## Format diff --git a/docs/measurements/05-qr.md b/docs/measurements/05-qr.md new file mode 100644 index 0000000..224e07f --- /dev/null +++ b/docs/measurements/05-qr.md @@ -0,0 +1,244 @@ +# 05 — Reading a QR code off the page an agent is stuck on + +**What was measured, and why.** A device-change check — reCAPTCHA's +scan-to-verify, a WhatsApp Web login, an authenticator enrolment — draws a QR +code and asks for a phone. handraise puts a human on a phone in front of that +page, and a phone cannot scan its own screen, so today it takes a second +device. Passing the link through instead needs three facts settled before any +code is written: can Chromium decode the code itself, is the live cast frame +good enough to decode from, and what does a scan cost. Measured 2026-09-02, +against a Solari cloud browser (Chrome 151, Linux x86_64) and local Chromium. + +Reproduce with: + +``` +bun --env-file=.env scripts/measure-qr-decode.ts --recaptcha # A, one browser session +bun scripts/measure-qr-decode.ts --local # B, no API key needed +``` + +**Short answer.** `BarcodeDetector` does not exist in Solari's Chromium, so the +decode happens in the agent process. A fresh full-resolution `page.screenshot()` +decodes a symbol the live cast frame cannot, and costs 293 ms end to end. Two +codes on one screen defeat the decoder outright unless the image is tiled. +A code the page resampled can be sharp and still not decode, which needs a +second look at 2x. reCAPTCHA's own demo never showed the QR variant, so that +specific site remains untested — see §7. + +--- + +## 1. `BarcodeDetector` in Solari's Chromium: absent + +```json +{"event":"barcode_detector","present":false, + "ua":"Mozilla/5.0 (X11; Linux x86_64) … Chrome/151.0.0.0 Safari/537.36"} +``` + +Checked with `stealth: true` and `stealth: false`; absent in both, and +`HeadlessChrome/151.0.7922.34` without stealth. This is the expected outcome — +Chromium implements the Shape Detection API on top of platform frameworks +(macOS, Android, ChromeOS) and ships nothing for Linux — but it was worth one +API call, because a positive would have meant zero new dependencies. + +**Consequence:** the decode is a `jsqr` call in the agent process, fed by a PNG +decoder written against `node:zlib` (`src/core/png.ts`). One new runtime +dependency, pure JavaScript, no native build. + +## 2. Screenshot and decode: 293 ms p50 + +Ten runs, 1280×800 viewport, a 200-character payload drawn at 420 CSS px +(version 10, 57 modules, ~7 px per module): + +| | p50 | samples (ms) | +|---|---|---| +| `page.screenshot({ type: "png" })` | **239 ms** | 455, 232, 240, 234, 319, 252, 235, 238, 246, 232 | +| decode (PNG unfilter + `jsQR`) | **54.5 ms** | 92, 45, 49, 56, 46, 58, 53, 56, 57, 44 | +| total | **293.5 ms** | | + +The screenshot is the cost, and it is network: a CDP round trip from Germany to +the session's region. The decode is local and cheap enough to ignore. The first +sample of each column is a warm-up (455 ms, 92 ms) and is in the median like +everything else. + +Screenshot size: 43.7 KB PNG. Decoded payload: 200 characters, exactly. + +**Consequence:** a scan feels immediate on a phone, and there is no case for +scanning speculatively. It is rate-limited to one per 2 s in the core. + +## 3. The cast frame is not good enough, and this is where it stops + +The phone is already looking at a picture of the page: the screencast frame, at +`DEFAULT_PROFILE` — 800 px wide, JPEG quality 60. Decoding *that* would cost no +screenshot at all. It was tested against the real thing rather than an +approximation: the frame comes from this repo's own `startFramePump`, and it is +handed back to Chromium to be decoded by Chromium's own JPEG decoder before +`jsQR` sees the pixels. + +Same payload, drawn at six sizes: + +| symbol on the page | from `page.screenshot()` (1280 px PNG) | from the cast frame (800 px JPEG q60) | frame bytes | +|---|---|---|---| +| 420 px | decoded | decoded | 27.8 KB | +| 260 px | decoded | decoded | 16.2 KB | +| 180 px | decoded | **failed** | 11.8 KB | +| 150 px | decoded | decoded | 10.5 KB | +| 120 px | decoded | **failed** | 8.9 KB | +| 100 px | **failed** | **failed** | 7.8 KB | + +Note the non-monotonicity at 150 px. Below about 260 px the cast frame is at the +edge of what survives a 0.625 downscale plus quality-60 quantisation, and +whether a given size lands is a matter of how the module grid aligns to the +pixel grid — which is another way of saying it is luck. The screenshot has no +such band: it decodes down to 120 px and stops at 100 px. + +**Consequence:** every scan takes a fresh `page.screenshot({ type: "png" })`. +The 239 ms buys the difference between "works" and "works when the site happens +to draw its code large". + +**Known limit of the feature:** a symbol below roughly 120 CSS px, at +`deviceScaleFactor: 1`, will not decode. A human can scroll or zoom the remote +page and scan again, and the sheet says so when nothing is found. + +## 4. Two codes on one screen defeat the decoder + +`jsQR` locates a symbol by its three finder patterns. Two symbols put six in +front of it, and it does not pick one — it returns nothing at all: + +``` +1280×800 page, two 261 px codes, 80 px and 700 px from the left + whole image -> null + four 60% corners -> both codes, 63 ms +``` + +Reading the whole image and then painting out the symbol just found (which is +how a second code is reached at all) only helps once the first has been found. +So when the whole-image pass comes back empty, `scanImage` looks again at four +overlapping corner tiles. That costs a second decode only on a pass that had +already failed, and it turns "no QR code found" on a page that visibly has two +into both of them. + +## 5. A sharp code that will not decode, and the second look that fixes it + +Found by the first live run of the e2e, which failed with "0 codes" on a page +whose QR was large, centred and perfectly sharp to the eye. The screenshot is +kept as `src/core/fixtures/qr-centred.png`; everything below is measured +against exactly those pixels. + +``` +whole 1280x800 image -> null +tight crop of the symbol -> decoded +same crop plus 12px padding -> null +2x nearest-neighbour -> decoded, 209 ms +3x nearest-neighbour -> decoded, 462 ms +``` + +A tight crop decodes and the same pixels with twelve more pixels of margin do +not. That rules out the image and points at `jsQR`'s binarizer, which +thresholds in fixed 8x8 blocks: the page drew a 534-pixel image at 420 CSS +pixels, so a module is 4.9 pixels wide and the block grid straddles the module +boundaries. Shifting the crop shifts that alignment, which is why padding +changes the answer. Magnifying puts about ten pixels under each module and the +blocks line up again — nearest-neighbour, so not one new pixel of information. + +Tiling was tried first and rejected as a fix for this: four 60% corners miss a +symbol that straddles the centre, and the geometries that happened to work +(0.6 centre, 0.7 bottom-left) did so by the same alignment luck, with 0.5 and +0.8 finding nothing. Tuning tile sizes would have been fitting to one image. + +**Consequence:** `scanImage` looks three times, and the second and third run +only when the one before found nothing — the image as it came, the image at 2x, +then the four corners for the two-code case in §4. A page with no code at all +therefore costs all three, about 320 ms of CPU against a 239 ms screenshot. + +**This is why the live e2e exists.** Every offline test passed, the decoder read +a real cloud-browser screenshot, the phone UI worked in real Chromium, and the +feature did not work. + +## 6. What a scan costs the agent's event loop + +The decode is pure synchronous CPU, and it shares a loop with the frame pump, +the human's handback, the timeout and the browser's disconnect. Measured on a +3840x2160 screenshot — a 4K viewport, or a 1920x1080 one at device scale 2, and +the largest input a scan can be handed — with no code in it, so the whole +three-look ladder runs. The probe is a 5 ms heartbeat; the number that matters +is the longest it was kept waiting. + +``` +bun scripts/measure-qr-block.ts + +{"event":"input","size":"3840x2160","megapixels":8.3,"pngBytes":29369} +{"event":"main_thread","wallMs":2137,"stallMs":2132,"ticks":0} +{"event":"worker_first","wallMs":2102,"stallMs":4,"ticks":376} +{"event":"worker_warm","wallMs":2027,"stallMs":1,"ticks":362} +``` + +**On the agent's own loop, the heartbeat did not run once.** Not a late tick — +zero ticks in 2.1 seconds. Anything the handoff had to answer in that window +waited the whole of it, and a human pressing Scan QR repeatedly could keep it +there. + +**On a worker, the same decode costs the loop one millisecond.** The wall time +is unchanged, because the work is the same work; what changed is who waits for +it. The worker's own startup does not show up next to two seconds of decode, +and a handoff pays it once. + +So the decode runs in `dist/qr-worker.js`, on a thread started at the first +scan of a handoff and terminated when it settles. The six-second deadline in +`createQrScanner` is the other half of that: terminating is the only lever +there is over a worker that has stopped answering, which is precisely why the +work is over there. + +### Where the cost actually is, and what the caps had to become + +The first version of this shipped a 3 s deadline against a 33 MP cap, and the +two were picked independently. They did not fit. Measured across sizes, warm +worker, with the whole three-look ladder running (no code in the image): + +| screenshot | megapixels | 2x retry runs | decode | +|---|---|---|---| +| 1280x800 | 1.0 | yes | 307 ms | +| 3840x2160 | 8.3 | yes | 2082 ms | +| 4000x2500 | 10.0 | yes | 2795 ms | +| 4472x2236 | 10.0 | yes | 2901 ms | +| 3162x3162 | 10.0 | yes | **3688 ms** | +| 5000x4800 | 24.0 | no | 1215 ms | +| 6000x4000 | 24.0 | no | 1175 ms | + +**The cost is not the pixel count, it is whether the 2x pass runs.** A 24 MP +image too large to magnify decodes in 1.2 s; a 10 MP one that magnifies to +40 MP takes three times as long. At the old budget a 10 MP screenshot — a +legitimate one — blew the 3 s deadline outright and was killed on the way to an +answer it already had, which the phone would then have reported as "the agent +didn't answer". + +So the three numbers are now derived from one another rather than chosen +separately: + +- `MAX_SCAN_PIXELS` is **34 MP** — exactly what a 4K screenshot needs to + magnify (8.3 MP becomes 33.2) and nothing more. +- the worst input the caps admit is therefore that 4K screenshot, at **2.1 s**. +- `DECODE_TIMEOUT_MS` is **6 s**, near three times that, and 5 s of screenshot + plus 6 s of decode still fits inside the phone's own 12 s wait. +- `MAX_PIXELS` is **24 MP**, which the no-magnify path finishes in 1.2 s. A 4K + viewport at device scale 2 is 33.2 MP and is refused; png.ts says so. + +## 7. reCAPTCHA: not reproducible, and therefore not claimed + +The feature exists because of reCAPTCHA's scan-to-verify variant. It could not +be reproduced: + +```json +{"event":"recaptcha_demo","url":"https://www.google.com/recaptcha/api2/demo", + "codes":0,"first":""} +``` + +The demo at `google.com/recaptcha/api2/demo`, loaded in a Solari browser with +stealth on and again without it, served the checkbox and an image challenge. The +device-change variant is served at Google's discretion, on signals nobody +outside Google controls, and no amount of retrying makes it appear on demand. + +**What this means, stated plainly:** the mechanism is proven end to end against +a site that behaves the way reCAPTCHA's variant behaves — the test app's `/qr` +page, driven by the live e2e — and it is **untested against reCAPTCHA itself**. +The README says the same thing. If it turns out reCAPTCHA's code is drawn below +the ~120 px floor in §3, or is not a link at all, this will need revisiting with +a real sample. diff --git a/docs/measurements/README.md b/docs/measurements/README.md index 15e1a14..5aabf53 100644 --- a/docs/measurements/README.md +++ b/docs/measurements/README.md @@ -1,9 +1,9 @@ # Platform measurements Four questions had to be answered against the live Solari API before handraise -could be designed at all. These are the write-ups, kept as evidence: the -numbers are unchanged from the day they were taken, and the ADRs in -[`../adr/`](../adr/) cite them. +could be designed at all, and one more before the QR passthrough was built. +These are the write-ups, kept as evidence: the numbers are unchanged from the +day they were taken, and the ADRs in [`../adr/`](../adr/) cite them. | # | What was measured | When | How | |---|---|---|---| @@ -11,11 +11,14 @@ numbers are unchanged from the day they were taken, and the ADRs in | [02](02-cdp-screencast.md) | CDP screencast over the SDK: framerate, frame size, bandwidth, delivery lag | 2026-09-01 | Four runs, three browser sessions, six scenarios | | [03](03-cdp-input-injection.md) | CDP input injection: mouse, text, keys, touch, scroll, `isTrusted`, coordinate mapping | 2026-09-01 | Every result read back from page state, never from "no error was thrown" | | [04](04-browser-session-lifetime.md) | Whether a browser session survives a multi-minute human pause | 2026-09-01 | Six browser sessions and five sandboxes, idle vs. pinged vs. streaming | +| [05](05-qr.md) | Reading a QR code off the page: `BarcodeDetector`, screenshot vs. cast frame, decode latency | 2026-09-02 | One cloud browser session and local Chromium; reproducible with `scripts/measure-qr-decode.ts` | **04 is the one to read first.** It is why the default wait is five minutes, why there is no keep-alive pinger, why `disconnected` is an outcome rather than an exception, and why `storageState` is captured on handback. -The probe scripts behind these documents were throwaway experiments and are not -carried in the tree; the repository history has them. Timing benchmarks of the +The probe scripts behind 01-04 were throwaway experiments and are not carried +in the tree; the repository history has them. 05 is reproducible from +[`scripts/measure-qr-decode.ts`](../../scripts/measure-qr-decode.ts), which also +regenerates the decoder's test fixture. Timing benchmarks of the shipped library live in [`../../benchmarks/`](../../benchmarks/). diff --git a/e2e/handoff.e2e.ts b/e2e/handoff.e2e.ts index 17d26a6..0a6c8e0 100644 --- a/e2e/handoff.e2e.ts +++ b/e2e/handoff.e2e.ts @@ -26,7 +26,7 @@ import type { Page } from "playwright-core" import type { HandoffEvent } from "../src/events" import { raiseHand } from "../src/index" -import { startTestApp } from "../test-app/deploy" +import { previewPath, startTestApp } from "../test-app/deploy" import { msUntilNextStep, totp } from "../test-app/totp" import { openHandoffPage } from "./human-sim" @@ -243,6 +243,111 @@ try { ) await relayGone.text() + // --- QR passthrough: the code on the page, opened on the phone --------- + // + // The device-change check, which the human on a phone cannot answer by + // scanning their own screen. The agent reads the code off a full-resolution + // screenshot and hands the human the link; the human opens it, and the site + // is satisfied on a device that has never seen it before. + const qrAt = Date.now() + await page.goto(previewPath(app.url, "/qr"), { + waitUntil: "domcontentloaded", + timeout: 30_000, + }) + await page.waitForSelector('[data-testid="qr-code"]', { timeout: 15_000 }) + + // The selector only says the element is there. A scan that finds nothing is + // then two different bugs — a code that never drew, or a decoder that could + // not read it — and this is what tells them apart. + const drawn = await page.evaluate(() => { + const image = document.querySelector("img") + if (!image) return null + const rect = image.getBoundingClientRect() + return { + complete: image.complete, + natural: image.naturalWidth, + css: Math.round(rect.width), + src: image.src.length, + } + }) + log("qr_page", drawn ?? { drawn: false }) + check( + (drawn?.natural ?? 0) > 0, + `the code is drawn on the page (${JSON.stringify(drawn)})`, + ) + + let qrUrl = "" + let qrEvent: HandoffEvent | undefined + const scanning = raiseHand(page, { + reason: "Aurora Bank wants this code scanned with your phone", + qr: false, + timeoutMs: 60_000, + onUrl: (url) => { + qrUrl = url + }, + onEvent: (raised) => { + qrEvent = raised + }, + }) + pending = scanning + + while (qrUrl === "") await Bun.sleep(50) + const scanner = await openHandoffPage(qrUrl) + await scanner.waitForFrame() + + const scanAt = Date.now() + const links = await scanner.scanqr() + timings.qrScanMs = Date.now() - scanAt + log("qr_scanned", { + ms: timings.qrScanMs, + count: links.length, + kind: links[0]?.kind, + }) + if (links.length === 0) { + // Keep the pixels the agent was looking at. Reading a failure off a + // screenshot beats guessing at it from a count. + const evidence = "/tmp/handraise-qr-e2e-failure.png" + await Bun.write(evidence, await page.screenshot({ type: "png" })) + log("qr_evidence", { path: evidence }) + } + check( + links.length === 1, + `the agent found exactly one code (${links.length})`, + ) + check( + links[0]?.text === app.verifyUrl, + "the link the human got is the one inside the code on the page", + ) + check(links[0]?.kind === "url", "an https link is offered as openable") + + // The human "opens" it. A phone, not this browser: no session cookie, no + // preview cookie, nothing but the link itself. + const visited = await fetch(links[0]?.text ?? "", { cache: "no-store" }) + const visitedBody = await visited.text() + check(visited.status === 200, `the link opens (${visited.status})`) + check( + visitedBody.includes('data-testid="verified"'), + "opening it reached the confirmation page", + ) + + await scanner.handback() + const scanned = await scanning + pending = null + timings.qrCaseMs = Date.now() - qrAt + log("qr_done", { + outcome: scanned.outcome, + scans: qrEvent?.qrScans, + hits: qrEvent?.qrHits, + ms: timings.qrCaseMs, + }) + check(scanned.outcome === "resolved", "the QR handoff resolved") + check( + qrEvent?.qrScans === 1, + `the wide event counts one scan (${qrEvent?.qrScans})`, + ) + check(qrEvent?.qrHits === 1, `and one hit (${qrEvent?.qrHits})`) + await scanner.close() + // --- Approval: the human answers a question, and drives nothing -------- // // The other half of the product. No screencast, no input path: one diff --git a/e2e/human-sim.ts b/e2e/human-sim.ts index 1c63ac1..7a88603 100644 --- a/e2e/human-sim.ts +++ b/e2e/human-sim.ts @@ -12,6 +12,7 @@ */ import WebSocket from "ws" +import type { ScannedLink } from "../src/core/qr-scan" import { type AgentToHuman, type FrameMeta, @@ -42,7 +43,11 @@ export interface SimulatedHuman { action(): string /** How the agent said the handoff ended, if it has. */ ending(): HandoffOutcome | null + /** The links from the newest answered scan, or null before the first one. */ + links(): ScannedLink[] | null waitForFrame(timeoutMs?: number): Promise + /** Ask the agent to read the QR codes on the page, and wait for its answer. */ + scanqr(timeoutMs?: number): Promise tap(fx: number, fy: number): Promise /** Type one character per message, the way the mobile UI does. */ type(text: string, delayMs?: number): Promise @@ -101,6 +106,7 @@ export async function openHandoffPage( let reason = "" let action = "" let ending: HandoffOutcome | null = null + let links: ScannedLink[] | null = null const waiters: (() => void)[] = [] socket.on("message", (data: Buffer) => { @@ -117,6 +123,7 @@ export async function openHandoffPage( reason = message.reason action = message.action ?? "" } + if (message.type === "links") links = message.links if (message.type === "ended") ending = message.outcome }) @@ -146,6 +153,7 @@ export async function openHandoffPage( reason: () => reason, action: () => action, ending: () => ending, + links: () => links, async waitForFrame(timeoutMs = 30_000) { const deadline = Date.now() + timeoutMs @@ -166,6 +174,19 @@ export async function openHandoffPage( tap: (fx, fy) => send({ type: "tap", fx, fy }), + async scanqr(timeoutMs = 30_000) { + links = null + await send({ type: "scanqr" }) + const deadline = Date.now() + timeoutMs + while (links === null) { + if (Date.now() > deadline) { + throw new Error(`no links answer within ${timeoutMs}ms`) + } + await Bun.sleep(100) + } + return links + }, + async type(text, delayMs = 60) { for (const ch of text) { await send({ type: "char", ch }) diff --git a/e2e/ui.spec.ts b/e2e/ui.spec.ts index 586d156..10950ce 100644 --- a/e2e/ui.spec.ts +++ b/e2e/ui.spec.ts @@ -26,6 +26,7 @@ import { fileURLToPath } from "node:url" import { type Browser, chromium, type Page } from "playwright-core" import WebSocket from "ws" +import { NEVER_OPENABLE } from "../src/core/qr-fixtures" import type { FocusRect, FrameMeta, @@ -81,6 +82,8 @@ interface RelayLog { interface AgentClient { send(message: RelayMessage): void + /** Put bytes on the wire that the protocol has no way to describe. */ + sendRaw(text: string): void next(): Promise /** Every message this socket has seen, in order. `next()` never consumes it, * so a test can assert that something was sent *exactly once*. */ @@ -188,6 +191,9 @@ async function connectAgent(port: number): Promise { send(message) { socket.send(JSON.stringify(message)) }, + sendRaw(text) { + socket.send(text) + }, received, next() { const queued = inbox.shift() @@ -618,7 +624,7 @@ test("the clear key is offered only while a remote field is focused", async () = expect(consoleErrors).toEqual([]) }) -const KEY_IDS = ["#key-back", "#key-tab", "#key-enter", "#key-clear"] +const KEY_IDS = ["#key-back", "#key-tab", "#key-enter", "#key-clear", "#key-qr"] /** Every key button's box, in the order the ids are given. */ async function keyBoxes(): Promise { @@ -1387,3 +1393,402 @@ test("the reconnect loop gives up flushing once its deadline passes", async () = ) expect(unexpected).toEqual([]) }, 20000) + +// --- QR passthrough: the button, the sheet, and what it refuses to open ---- + +const QR_LINK = "https://verify.example.com/device?token=abc123" + +/** + * Wait for the result sheet to be on screen (or gone). + * + * When it is coming in, wait for the card to have finished rising as well: it + * starts a `translateY(100%)` below the fold, so between the `hidden` flip and + * the end of the transition its contents are outside the viewport and + * `elementFromPoint` over them answers null. + */ +async function waitForSheet(visible: boolean): Promise { + await page.waitForFunction((want: boolean) => { + const sheet = document.getElementById("sheet") + if (!sheet || !sheet.hidden !== want) return false + if (!want) return true + const card = document.getElementById("sheet-card") + return card ? card.getBoundingClientRect().bottom <= innerHeight + 1 : false + }, visible) +} + +/** The text of every link card in the sheet, in order. */ +function sheetTexts(): Promise { + return page + .locator("#sheet-links .link-text") + .allTextContents() + .then((texts) => texts.map((text) => text.trim())) +} + +test("the scan button is offered in a takeover and not in an approval", async () => { + await showFrame() + expect(await page.locator("#key-qr").isVisible()).toBe(true) + expect(await page.locator("#key-qr").isEnabled()).toBe(true) + + // The whole input bar is gone in an approval: the human is answering a + // question about one screenshot, and there is no live page to scan. + await reopenFixture("approval") + agent.send({ type: "frame", data: frameData, meta: META }) + await page.waitForTimeout(150) + expect(await page.locator("#key-qr").isVisible()).toBe(false) + + // The page will not even put it on the wire: `scanqr` is not in an approval's + // vocabulary. (The relay refuses it too — relay.test.ts "approval mode drops + // every takeover message the human sends" — because a hidden control is not + // a restriction.) + await page.evaluate(() => { + document.getElementById("key-qr")?.click() + }) + await page.waitForTimeout(200) + expect(agent.received.some((message) => message.type === "scanqr")).toBe( + false, + ) + expect(consoleErrors).toEqual([]) +}) + +test("pressing scan asks the agent once and waits for the answer", async () => { + await showFrame() + + await page.locator("#key-qr").click() + await page.waitForFunction(() => { + const button = document.getElementById("key-qr") + return button instanceof HTMLButtonElement && button.disabled + }) + const scans = agent.received.filter((message) => message.type === "scanqr") + expect(scans).toHaveLength(1) + expect(await page.locator("#hint").textContent()).toBe("Reading the page…") + + // A second press while the first is in flight must not reach the agent: the + // core would drop it anyway, and a dropped scan is an answer that never comes. + await page.locator("#key-qr").click({ force: true }) + await page.waitForTimeout(100) + expect( + agent.received.filter((message) => message.type === "scanqr"), + ).toHaveLength(1) + + agent.send({ + type: "links", + links: [{ text: QR_LINK, kind: "url" }], + source: "qr", + }) + await waitForSheet(true) + expect(await page.locator("#key-qr").isEnabled()).toBe(true) + expect(consoleErrors).toEqual([]) +}) + +test("the sheet shows the link and opens it in a new tab, never in this one", async () => { + await showFrame() + agent.send({ + type: "links", + links: [{ text: QR_LINK, kind: "url" }], + source: "qr", + }) + await waitForSheet(true) + + expect(await sheetTexts()).toEqual([QR_LINK]) + const open = page.locator("#sheet-links a.link-action") + expect(await open.isVisible()).toBe(true) + expect(await open.getAttribute("href")).toBe(QR_LINK) + // This tab is holding a live handoff. The opened site must not get a handle + // on it, and must not be told the handoff URL it came from. + expect(await open.getAttribute("target")).toBe("_blank") + expect(await open.getAttribute("rel")).toBe("noopener noreferrer") + + // Copy is offered whatever the link is, and the sheet is dismissible. + expect( + await page.locator("#sheet-links button.link-action").textContent(), + ).toBe("Copy") + await page.locator("#sheet-close").click() + await waitForSheet(false) + expect(consoleErrors).toEqual([]) +}) + +test("a scan that found nothing says so rather than showing an empty sheet", async () => { + await showFrame() + await page.locator("#key-qr").click() + agent.send({ type: "links", links: [], source: "qr" }) + await waitForSheet(true) + + expect(await page.locator("#sheet-title").textContent()).toBe( + "No QR code found", + ) + expect(await sheetTexts()).toEqual([]) + expect(await page.locator("#sheet-links .empty").textContent()).toContain( + "Nothing on this screen decoded", + ) + // And the button is usable again, or the human cannot try after scrolling. + expect(await page.locator("#key-qr").isEnabled()).toBe(true) + expect(consoleErrors).toEqual([]) +}) + +test("a payload the page may not open gets no anchor, whatever the agent called it", async () => { + await showFrame() + // Every one of these arrives labelled `kind: "url"`, which is the lie the + // page has to survive: the handoff URL is a bearer credential and the socket + // behind it is reachable from any HTTP client, so this side applies the whole + // rule again instead of trusting the label. The list is the core's own, so + // the two locks cannot drift into checking different things. + agent.send({ + type: "links", + links: NEVER_OPENABLE.map((text) => ({ text, kind: "url" as const })), + source: "qr", + }) + await waitForSheet(true) + + expect(await sheetTexts()).toEqual([...NEVER_OPENABLE]) + // No anchor at all — not a disabled one, and not one with a neutered href. + expect(await page.locator("#sheet-links a").count()).toBe(0) + expect(await page.locator("#sheet-links button.link-action").count()).toBe( + NEVER_OPENABLE.length, + ) + expect( + await page.locator("#sheet-links .link-note").first().textContent(), + ).toContain("Not a link this page will open") + expect(consoleErrors).toEqual([]) +}) + +test("an openable link is shown as the address it actually opens", async () => { + await showFrame() + // The first character is a Cyrillic a. The eye reads apple.com and the + // browser goes to xn--pple-43d.com, so showing the payload beside an anchor + // that resolves it shows the human one address and opens another. (A payload + // carrying a bidi override never gets this far — it is refused outright, and + // `NEVER_OPENABLE` covers that.) + const homograph = "https://аpple.com/verify?token=abc" + const resolved = new URL(homograph).href + expect(resolved).not.toBe(homograph) + expect(resolved).toContain("xn--pple-43d.com") + + agent.send({ + type: "links", + links: [{ text: homograph, kind: "url" }], + source: "qr", + }) + await waitForSheet(true) + + // Shown, anchored and copied: one string, and it is the resolved one. + expect(await sheetTexts()).toEqual([resolved]) + expect( + await page.locator("#sheet-links a.link-action").getAttribute("href"), + ).toBe(resolved) + expect( + await page.locator("#sheet-links .link-note").first().textContent(), + ).toContain("wrote this address differently") + expect(consoleErrors).toEqual([]) +}) + +test("a dialer string and an authenticator secret are named, not opened", async () => { + await showFrame() + // "Not a link" says nothing useful about either of these, and both are + // things a human should hand to an app deliberately rather than in one tap + // from a page nobody vetted. + agent.send({ + type: "links", + links: [ + { text: "tel:*21*1234567890%23", kind: "url" }, + { + text: "otpauth://totp/Example:ada?secret=JBSWY3DPEHPK3PXP", + kind: "url", + }, + ], + source: "qr", + }) + await waitForSheet(true) + + expect(await page.locator("#sheet-links a").count()).toBe(0) + const notes = await page.locator("#sheet-links .link-note").allTextContents() + expect(notes[0]).toContain("Phone number") + expect(notes[1]).toContain("Authenticator secret") + expect(consoleErrors).toEqual([]) +}) + +test("the host of an openable link is the loud part of it", async () => { + await showFrame() + agent.send({ + type: "links", + links: [{ text: QR_LINK, kind: "url" }], + source: "qr", + }) + await waitForSheet(true) + + // The one question a human answers before tapping Open is whose site this + // is, and on a 390px screen the host is otherwise a few characters lost in + // a token. + const host = page.locator("#sheet-links .link-host") + expect(await host.textContent()).toBe("verify.example.com") + const weights = await page.evaluate(() => { + const loud = document.querySelector("#sheet-links .link-host") + const rest = document.querySelector("#sheet-links .link-text > span") + if (!loud || !rest) return null + return { + loud: getComputedStyle(loud).fontWeight, + quiet: getComputedStyle(rest).color, + loudColour: getComputedStyle(loud).color, + } + }) + expect(Number(weights?.loud)).toBeGreaterThanOrEqual(600) + expect(weights?.quiet).not.toBe(weights?.loudColour) + expect(consoleErrors).toEqual([]) +}) + +test("an ordinary link is shown verbatim, with no note about it", async () => { + await showFrame() + agent.send({ + type: "links", + links: [{ text: QR_LINK, kind: "url" }], + source: "qr", + }) + await waitForSheet(true) + + expect(await sheetTexts()).toEqual([QR_LINK]) + expect(await page.locator("#sheet-links .link-note").count()).toBe(0) + expect(consoleErrors).toEqual([]) +}) + +test("normalising a link is not the same as changing it", async () => { + await showFrame() + // Both of these come back from the URL parser as a different string — a + // trailing slash appears, a capital is lowered — and neither is a deception. + // A bare domain is one of the commonest shapes a QR code has, and a warning + // that fires on it is a warning the human learns to tap past, which is + // exactly when the homograph case needs it to land. + agent.send({ + type: "links", + links: [ + { text: "https://example.com", kind: "url" }, + { text: "HTTPS://Example.COM/Path", kind: "url" }, + ], + source: "qr", + }) + await waitForSheet(true) + + expect(await sheetTexts()).toEqual([ + "https://example.com/", + "https://example.com/Path", + ]) + expect(await page.locator("#sheet-links .link-note").count()).toBe(0) + expect(await page.locator("#sheet-links a.link-action").count()).toBe(2) + expect(consoleErrors).toEqual([]) +}) + +test("the result sheet stays reachable after the handoff ends", async () => { + await showFrame() + agent.send({ + type: "links", + links: [{ text: QR_LINK, kind: "url" }], + source: "qr", + }) + await waitForSheet(true) + + // The feature's own happy path: the human reads the link and the handoff ends + // before they tap Open — they hand back, or it times out. The ending overlay + // is opaque and covers the whole screen, so if it wins the stacking order the + // link is gone, with the button disabled and no way to scan again. + agent.send({ type: "ended", outcome: "resolved" }) + await waitForOverlay(page) + + const onTop = await page.evaluate(() => { + const anchor = document.querySelector("#sheet-links a.link-action") + if (!anchor) return "no anchor" + const box = anchor.getBoundingClientRect() + const hit = document.elementFromPoint( + box.x + box.width / 2, + box.y + box.height / 2, + ) + return hit?.closest("#sheet") ? "sheet" : (hit?.id ?? hit?.tagName ?? "?") + }) + expect(onTop).toBe("sheet") + expect(consoleErrors).toEqual([]) +}) + +test("a malformed links message neither throws nor wedges the button", async () => { + await showFrame() + await page.locator("#key-qr").click() + await page.waitForFunction(() => { + const button = document.getElementById("key-qr") + return button instanceof HTMLButtonElement && button.disabled + }) + + // A null, a number and an object with no text. This used to throw out of the + // message handler, which skipped the code that releases the button — leaving + // it dead for the full twelve-second deadline, with no sheet and no reason. + // Sent as bytes rather than as a typed message: the protocol has no way to + // describe this, and casting one into shape would be the same lie the page + // is being tested against. + agent.sendRaw( + '{"type":"links","links":[null,5,{"kind":"url"}],"source":"qr"}', + ) + await waitForSheet(true) + + expect(await sheetTexts()).toEqual([]) + expect(await page.locator("#sheet-title").textContent()).toBe( + "No QR code found", + ) + expect(await page.locator("#key-qr").isEnabled()).toBe(true) + expect(consoleErrors).toEqual([]) +}) + +test("a QR payload reaches the sheet as text, never as markup", async () => { + await showFrame() + const payload = '' + agent.send({ + type: "links", + links: [{ text: payload, kind: "text" }], + source: "qr", + }) + await waitForSheet(true) + + expect(await sheetTexts()).toEqual([payload]) + expect(await page.locator("#sheet-links img").count()).toBe(0) + expect(consoleErrors).toEqual([]) +}) + +test("two codes are both listed, and the sheet says there are two", async () => { + await showFrame() + agent.send({ + type: "links", + links: [ + { text: QR_LINK, kind: "url" }, + { text: "WIFI:S:GuestNet;T:WPA;P:hunter2;;", kind: "text" }, + ], + source: "qr", + }) + await waitForSheet(true) + + expect(await page.locator("#sheet-title").textContent()).toBe( + "2 codes on the page", + ) + expect(await sheetTexts()).toEqual([ + QR_LINK, + "WIFI:S:GuestNet;T:WPA;P:hunter2;;", + ]) + expect(await page.locator("#sheet-links a").count()).toBe(1) + expect(consoleErrors).toEqual([]) +}) + +test("a scan the agent never answers releases the button and says why", async () => { + await reopenFixture("takeover", false, true) + await showFrame() + await page.locator("#key-qr").click() + await page.waitForFunction(() => { + const button = document.getElementById("key-qr") + return button instanceof HTMLButtonElement && button.disabled + }) + + // The agent drops a scan that came too soon, or has gone away entirely. + // Without a deadline the button would stay dead for the rest of the session. + await page.clock.fastForward(12_000) + await page.waitForFunction(() => { + const button = document.getElementById("key-qr") + return button instanceof HTMLButtonElement && !button.disabled + }) + expect(await page.locator("#hint").textContent()).toBe( + "The agent didn't answer — try again", + ) + expect(await page.locator("#sheet").isHidden()).toBe(true) + expect(consoleErrors).toEqual([]) +}) diff --git a/package.json b/package.json index 19178bc..104f68c 100644 --- a/package.json +++ b/package.json @@ -49,6 +49,7 @@ "dependencies": { "@solarisdk/browser": "^0.1.2", "@solarisdk/sdk": "^0.1.2", + "jsqr": "1.4.0", "qrcode-terminal": "^0.12.0", "ws": "^8.21.3" }, @@ -60,12 +61,14 @@ "@ai-sdk/openai-compatible": "^3.0.41", "@biomejs/biome": "^2.0.0", "@oxlint/plugins": "1.80.0", + "@types/qrcode": "^1.5.6", "@types/qrcode-terminal": "^0.12.2", "@types/ws": "^8.5.13", "ai": "^7.0.87", "bun-types": "^1.4.0", "oxlint": "1.80.0", "playwright-core": "^1.62.1", + "qrcode": "^1.5.4", "tsup": "^8.5.1", "typescript": "^5.7.2" }, diff --git a/scripts/dist-smoke.mjs b/scripts/dist-smoke.mjs index bf8305b..2723f64 100644 --- a/scripts/dist-smoke.mjs +++ b/scripts/dist-smoke.mjs @@ -1,10 +1,15 @@ // Runs the SHIPPED artifact under node — the consumer's runtime, not bun's. // Exists because a CJS-interop difference made the QR silently break in dist // while every bun-driven test stayed green. +import { readFileSync } from "node:fs" + const m = await import("../dist/index.js") const expected = [ "raiseHand", "handoffQr", + "scanQrLinks", + "createQrScanner", + "OPENABLE_SCHEMES", "consoleLogger", "quietLogger", "noopLogger", @@ -38,10 +43,37 @@ if ( const qr = m.handoffQr( `https://example.preview.getsolari.com/?pt_token=${"x".repeat(240)}`, ) -if (!qr || !qr.includes("▄")) { +if (!qr?.includes("▄")) { console.error("dist smoke: QR did not render under node") process.exit(1) } +// The other direction, and the same trap: `jsqr` is a CommonJS UMD bundle, so +// its default import is exactly the shape that broke `qrcode-terminal` in dist +// while bun stayed green. Decode a real screenshot to prove it survived. +const shot = readFileSync( + new URL("../src/core/fixtures/qr-page.png", import.meta.url), +) +const links = m.scanQrLinks(shot) +if (links.length !== 1 || links[0].kind !== "url") { + console.error( + "dist smoke: the QR decoder did not read the fixture under node", + ) + process.exit(1) +} + +// The worker is a second build entry loaded by URL at runtime, so it is the +// one part of this package that a bundler can drop without anything failing to +// import. Under node, against dist/qr-worker.js, on a real screenshot. +const scanner = m.createQrScanner() +const offThread = await scanner.scan(shot) +await scanner.close() +if (offThread.length !== 1 || offThread[0].text !== links[0].text) { + console.error( + "dist smoke: the QR worker did not decode the fixture under node", + ) + process.exit(1) +} + console.log( - `dist smoke ok — ${expected.length} exports, QR ${qr.split("\n").length} rows`, + `dist smoke ok — ${expected.length} exports, QR ${qr.split("\n").length} rows, decoded ${links[0].text.length} chars on the main thread and on the worker`, ) diff --git a/scripts/measure-qr-block.ts b/scripts/measure-qr-block.ts new file mode 100644 index 0000000..7192938 --- /dev/null +++ b/scripts/measure-qr-block.ts @@ -0,0 +1,101 @@ +/** + * Measurement 05 §7: what a scan costs the agent's event loop. + * + * bun scripts/measure-qr-block.ts + * node --experimental-strip-types scripts/measure-qr-block.ts + * + * No API key and no browser: the input is a synthetic 3840x2160 screenshot, + * which is the largest thing a scan can be asked to decode in practice (a 4K + * viewport, or a 1920x1080 one at device scale 2). What is measured is not how + * long the decode takes — that is in §2 — but how long the loop cannot answer + * anything while it happens, before and after the work moved to a worker. + * + * The probe is a 5 ms interval that records how late each tick is. The largest + * gap is the answer: it is how long a handback, a timeout or a screencast + * frame would have waited. + */ +import { deflateSync } from "node:zlib" + +import { createQrScanner, scanQrLinks } from "../src/core/qr-scan" + +const WIDTH = 3840 +const HEIGHT = 2160 +const TICK_MS = 5 + +/** A white RGB PNG of the given size: no code in it, so the whole ladder runs. */ +function blankPng(width: number, height: number): Buffer { + const raw = Buffer.alloc(height * (width * 3 + 1), 0xff) + for (let y = 0; y < height; y++) raw[y * (width * 3 + 1)] = 0 + const chunk = (type: string, body: Buffer): Buffer => { + const head = Buffer.alloc(8) + head.writeUInt32BE(body.length, 0) + head.write(type, 4, "ascii") + return Buffer.concat([head, body, Buffer.alloc(4)]) + } + const ihdr = Buffer.alloc(13) + ihdr.writeUInt32BE(width, 0) + ihdr.writeUInt32BE(height, 4) + ihdr[8] = 8 + ihdr[9] = 2 + return Buffer.concat([ + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), + chunk("IHDR", ihdr), + chunk("IDAT", deflateSync(raw)), + chunk("IEND", Buffer.alloc(0)), + ]) +} + +/** Run `work`, and report the longest the 5 ms heartbeat was kept waiting. */ +async function longestStall(work: () => Promise): Promise<{ + wallMs: number + stallMs: number + ticks: number +}> { + let last = Date.now() + let stall = 0 + let ticks = 0 + const beat = setInterval(() => { + const now = Date.now() + stall = Math.max(stall, now - last - TICK_MS) + last = now + ticks += 1 + }, TICK_MS) + const started = Date.now() + await work() + // The last gap counts too, and on the blocking path it is the only one: + // an interval that never got to run recorded nothing at all. + const wallMs = Date.now() - started + stall = Math.max(stall, Date.now() - last - TICK_MS) + clearInterval(beat) + return { wallMs, stallMs: stall, ticks } +} + +const png = blankPng(WIDTH, HEIGHT) +console.log( + JSON.stringify({ + event: "input", + size: `${WIDTH}x${HEIGHT}`, + megapixels: Number(((WIDTH * HEIGHT) / 1e6).toFixed(1)), + pngBytes: png.length, + }), +) + +// Before: the decode on the agent's own loop, which is what shipped first. +const onLoop = await longestStall(async () => { + scanQrLinks(png) +}) +console.log(JSON.stringify({ event: "main_thread", ...onLoop })) + +// After: the same decode, on the worker. +const scanner = createQrScanner() +// The first scan pays the worker's startup; report it separately, because a +// handoff pays it once and every scan after this one does not. +const firstOnWorker = await longestStall(async () => { + await scanner.scan(png) +}) +console.log(JSON.stringify({ event: "worker_first", ...firstOnWorker })) +const onWorker = await longestStall(async () => { + await scanner.scan(png) +}) +console.log(JSON.stringify({ event: "worker_warm", ...onWorker })) +await scanner.close() diff --git a/scripts/measure-qr-decode.ts b/scripts/measure-qr-decode.ts new file mode 100644 index 0000000..18c96cb --- /dev/null +++ b/scripts/measure-qr-decode.ts @@ -0,0 +1,227 @@ +/** + * Measurement 05: reading a QR code off the page an agent is stuck on. + * + * bun --env-file=.env scripts/measure-qr-decode.ts + * bun --env-file=.env scripts/measure-qr-decode.ts --recaptcha + * bun scripts/measure-qr-decode.ts --local # part B only, no API key + * + * Answers the questions plan 04 asked before the feature was built: + * + * A, on a real Solari cloud browser: + * 1. Is `BarcodeDetector` available? If it were, the decode could happen + * inside Chromium and cost no dependency. + * 2. What does a full-resolution `page.screenshot()` plus a decode cost, + * over ten runs? + * Also writes the unit tests' fixture — a real screenshot from a real + * cloud browser — to src/core/fixtures/qr-page.png. + * + * B, on local Chromium, because it needs no cloud and is deterministic: + * 3. Would the live cast frame do instead of a screenshot? The frame is + * produced by this repo's own `startFramePump` at `DEFAULT_PROFILE`, + * and decoded through Chromium's own JPEG decoder — not an + * approximation of one — at three symbol sizes. + * + * Results: docs/measurements/05-qr.md. + */ + +import { writeFileSync } from "node:fs" +import { fileURLToPath } from "node:url" +import { Solari } from "@solarisdk/browser" +import { chromium, type Page } from "playwright-core" +import QRCode from "qrcode" + +import { decodePng } from "../src/core/png" +import { scanImage, scanQrLinks } from "../src/core/qr-scan" +import { DEFAULT_PROFILE, startFramePump } from "../src/core/screencast" + +const RUNS = 10 +const VIEWPORT = { width: 1280, height: 800 } +/** The sizes a device-change prompt draws its code at, in CSS pixels. */ +const SYMBOL_SIZES = [420, 260, 180, 150, 120, 100] +const FIXTURE = fileURLToPath( + new URL("../src/core/fixtures/qr-page.png", import.meta.url), +) + +/** A payload the length of a real device-handoff link, so the symbol is dense. */ +const PAYLOAD = `https://verify.example.com/device?token=${"a1b2c3d4".repeat(20)}` + +function median(values: number[]): number { + const sorted = [...values].sort((a, b) => a - b) + const middle = Math.floor(sorted.length / 2) + if (sorted.length % 2 === 1) return sorted[middle] ?? 0 + return ((sorted[middle - 1] ?? 0) + (sorted[middle] ?? 0)) / 2 +} + +/** One wide JSON line per measurement, the way the e2e and the relay log. */ +type LogDetail = Record + +function log(event: string, detail: LogDetail): void { + console.log(JSON.stringify({ event, ...detail })) +} + +/** The page both parts measure: a heading, a sentence, and one QR code. */ +async function drawPage(page: Page, cssWidth: number): Promise { + const image = await QRCode.toDataURL(PAYLOAD, { scale: 6, margin: 2 }) + await page.setContent( + ` +

Confirm on another device

+

Scan this code with the phone you registered.

+ code + `, + ) +} + +/** What the symbol actually measures on the page, so the numbers can be read. */ +function symbolWidth(page: Page): Promise { + return page.evaluate( + () => document.getElementById("code")?.getBoundingClientRect().width ?? 0, + ) +} + +/** + * Hand a JPEG back to Chromium and take the pixels out as a PNG. + * + * The point is that Chromium decodes its own JPEG: a downscale written here + * would be an approximation of the cast frame, and this is the frame. + */ +async function jpegToPng(page: Page, base64: string): Promise { + const png = await page.evaluate(async (data: string) => { + const image = new Image() + image.src = `data:image/jpeg;base64,${data}` + await image.decode() + const canvas = document.createElement("canvas") + canvas.width = image.naturalWidth + canvas.height = image.naturalHeight + const context = canvas.getContext("2d") + if (!context) throw new Error("no 2d context") + context.drawImage(image, 0, 0) + return canvas.toDataURL("image/png") + }, base64) + return Buffer.from(png.replace(/^data:image\/png;base64,/, ""), "base64") +} + +/** One live cast frame at the profile the handoff actually uses. */ +async function castFrame(page: Page): Promise<{ data: string; width: number }> { + const cdp = await page.context().newCDPSession(page) + let resolve: (frame: { data: string; width: number }) => void = () => + undefined + const first = new Promise<{ data: string; width: number }>((done) => { + resolve = done + }) + const pump = await startFramePump( + cdp, + DEFAULT_PROFILE, + async (data, meta) => { + resolve({ data, width: meta.jpegWidth }) + }, + ) + const frame = await first + await pump.stop() + await cdp.detach().catch(() => undefined) + return frame +} + +// --------------------------------------------------------------- part A ---- + +if (!process.argv.includes("--local")) { + const apiKey = process.env.SOLARI_API_KEY + if (!apiKey) throw new Error("SOLARI_API_KEY missing — use --env-file=.env") + const solari = new Solari({ apiKey }) + try { + const browser = await solari.launch({ stealth: true }) + const context = browser.contexts()[0] ?? (await browser.newContext()) + const opened = context.pages()[0] ?? (await context.newPage()) + await opened.setViewportSize(VIEWPORT) + + log( + "barcode_detector", + await opened.evaluate(() => ({ + present: "BarcodeDetector" in window, + ua: navigator.userAgent, + })), + ) + + // SAFETY: `@solarisdk/browser` returns patchright-core's Page, whose + // runtime surface is the one used here (setContent, evaluate, screenshot); + // the two declarations differ only in optional-property variance. The same + // assertion the e2e makes. + const page = opened as Page + await drawPage(page, 420) + + const shotMs: number[] = [] + const decodeMs: number[] = [] + let decoded = "" + let bytes = 0 + for (let run = 0; run < RUNS; run++) { + const shotAt = Date.now() + const shot = await page.screenshot({ type: "png" }) + shotMs.push(Date.now() - shotAt) + bytes = shot.length + const decodeAt = Date.now() + decoded = scanQrLinks(shot)[0]?.text ?? "" + decodeMs.push(Date.now() - decodeAt) + if (run === 0) writeFileSync(FIXTURE, shot) + } + log("full_resolution", { + ok: decoded === PAYLOAD, + symbolPx: await symbolWidth(page), + bytes, + shotP50: median(shotMs), + decodeP50: median(decodeMs), + totalP50: median(shotMs) + median(decodeMs), + shot: shotMs.join(","), + decode: decodeMs.join(","), + }) + + if (process.argv.includes("--recaptcha")) { + await page.goto("https://www.google.com/recaptcha/api2/demo", { + waitUntil: "domcontentloaded", + timeout: 45_000, + }) + await new Promise((done) => setTimeout(done, 4_000)) + const shot = await page.screenshot({ type: "png", fullPage: true }) + const links = scanQrLinks(shot) + log("recaptcha_demo", { + url: page.url(), + codes: links.length, + first: links[0]?.text.slice(0, 120) ?? "", + }) + } + + await browser.close() + } finally { + await solari.close().catch(() => undefined) + } +} + +// --------------------------------------------------------------- part B ---- + +const local = await chromium.launch({ headless: true }) +try { + const page = await local.newPage({ + viewport: VIEWPORT, + deviceScaleFactor: 1, + }) + for (const size of SYMBOL_SIZES) { + await drawPage(page, size) + const symbolPx = await symbolWidth(page) + + const shot = await page.screenshot({ type: "png" }) + const fromScreenshot = scanImage(decodePng(shot))[0] === PAYLOAD + + const frame = await castFrame(page) + const fromCast = + scanImage(decodePng(await jpegToPng(page, frame.data)))[0] === PAYLOAD + + log("cast_versus_screenshot", { + symbolPx, + screenshot: `${VIEWPORT.width}px`, + fromScreenshot, + cast: `${frame.width}px jpeg q${DEFAULT_PROFILE.quality}`, + fromCast, + castBytes: Buffer.from(frame.data, "base64").length, + }) + } +} finally { + await local.close() +} diff --git a/src/core/fixtures/qr-centred.png b/src/core/fixtures/qr-centred.png new file mode 100644 index 0000000..28d8710 Binary files /dev/null and b/src/core/fixtures/qr-centred.png differ diff --git a/src/core/fixtures/qr-page.png b/src/core/fixtures/qr-page.png new file mode 100644 index 0000000..5a8f1b7 Binary files /dev/null and b/src/core/fixtures/qr-page.png differ diff --git a/src/core/handoff.test.ts b/src/core/handoff.test.ts index d00c582..bc8c4e6 100644 --- a/src/core/handoff.test.ts +++ b/src/core/handoff.test.ts @@ -15,6 +15,7 @@ import { spawn } from "node:child_process" import { readFileSync } from "node:fs" import type { AddressInfo } from "node:net" import { fileURLToPath } from "node:url" +import { deflateSync } from "node:zlib" import type { Browser, BrowserContext, CDPSession, Page } from "playwright-core" import WebSocket, { WebSocketServer } from "ws" @@ -167,11 +168,63 @@ const CLOSED_PORT = "http://127.0.0.1:1" const SAMPLE_JPEG = readFileSync( fileURLToPath(new URL("./fixtures/sample-frame.jpg", import.meta.url)), ) + +/** + * A real screenshot of a real page carrying a real QR code, so a `scanqr` is + * answered by the decoder rather than by a stub that always agrees. + * `page.screenshot({ type: "png" })` is what a scan asks for; an approval's one + * frame asks for a JPEG, and `fakePage` answers each with its own bytes. + */ +const QR_PAGE_PNG = readFileSync( + fileURLToPath(new URL("./fixtures/qr-page.png", import.meta.url)), +) +const QR_PAGE_LINK = `https://verify.example.com/device?token=${"a1b2c3d4".repeat(20)}` + +/** A white PNG: a page with nothing on it to find. */ +function blankPng(width: number, height: number): Buffer { + const raw = Buffer.alloc(height * (width * 3 + 1), 0xff) + for (let y = 0; y < height; y++) raw[y * (width * 3 + 1)] = 0 + const chunk = (type: string, body: Buffer): Buffer => { + const head = Buffer.alloc(8) + head.writeUInt32BE(body.length, 0) + head.write(type, 4, "ascii") + const crc = Buffer.alloc(4) + crc.writeUInt32BE(crc32(Buffer.concat([head.subarray(4), body])), 0) + return Buffer.concat([head, body, crc]) + } + const ihdr = Buffer.alloc(13) + ihdr.writeUInt32BE(width, 0) + ihdr.writeUInt32BE(height, 4) + ihdr[8] = 8 + ihdr[9] = 2 + return Buffer.concat([ + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), + chunk("IHDR", ihdr), + chunk("IDAT", deflateSync(raw)), + chunk("IEND", Buffer.alloc(0)), + ]) +} + +function crc32(bytes: Buffer): number { + let crc = 0xffffffff + for (const byte of bytes) { + crc ^= byte + for (let bit = 0; bit < 8; bit++) { + crc = crc & 1 ? (crc >>> 1) ^ 0xedb88320 : crc >>> 1 + } + } + return (crc ^ 0xffffffff) >>> 0 +} + +const BLANK_PNG = blankPng(64, 40) const VIEWPORT = { width: 1280, height: 800 } /** CDP sessions opened on the fake page since the last reset. */ let cdpSessions = 0 +/** Screenshots the newest `fakePage()` has been asked for. */ +let screenshots = 0 + /** Kills the browser session behind the newest `fakePage()`. */ let killSession: () => void = () => undefined @@ -185,8 +238,10 @@ function fakePage( cdp: CDPSession, screenshotDelayMs = 0, storageStateDelayMs = 0, + pngScreenshot: Buffer = QR_PAGE_PNG, ): Page { cdpSessions = 0 + screenshots = 0 let browser: Browser let connected = true const gone = new Set<() => void>() @@ -232,10 +287,12 @@ function fakePage( // A live page, which is what `raiseHand` checks before it starts anything. isClosed: () => false, // SAFETY: approval mode calls screenshot() for its one frame and reads the - // viewport for that frame's metadata; neither result is used as anything else. - screenshot: (async () => { + // viewport for that frame's metadata; a QR scan calls it for a PNG. Neither + // result is used as anything else. + screenshot: (async (options?: { type?: "png" | "jpeg" }) => { if (screenshotDelayMs > 0) await Bun.sleep(screenshotDelayMs) - return SAMPLE_JPEG + screenshots += 1 + return options?.type === "png" ? pngScreenshot : SAMPLE_JPEG }) as Page["screenshot"], viewportSize: () => VIEWPORT, // SAFETY: as the browser's, above — an unused chaining emitter. @@ -1618,3 +1675,226 @@ test("a logger whose methods reject does not break the handoff either", async () process.off("unhandledRejection", record) } }) + +test("a logger that throws does not break the handoff", async () => { + // `logger` is a public option and it is the caller's object: a pino instance + // over a closed transport throws. Every call handraise makes to it sits on a + // failure path or inside a promise callback, so a throw would either lose + // the outcome (the wide event is logged before it is returned) or reject a + // promise nobody awaits, and node ends the process for that. + const port = await startRelayProcess() + const human = await connectHuman(port) + const cdp = fakeCdp() + const down = (): never => { + throw new Error("logger is down (EPIPE)") + } + const hostile: Logger = { debug: down, info: down, warn: down, error: down } + const events: HandoffEvent[] = [] + + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + reason: "the logger is hostile", + logger: hostile, + onEvent: (event) => events.push(event), + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "hostile-logger", + relayColdStartMs: 5, + logger: hostile, + }) + + await until("the phone to connect", () => human.inbox.length >= 0) + human.send({ type: "handback" }) + + const end = await handoff + expect(end.outcome).toBe("resolved") + // The wide event still reaches the caller: `logger.info` throwing must not + // take `onEvent` with it. + expect(events).toHaveLength(1) +}) + +// --- QR passthrough -------------------------------------------------------- + +/** + * A takeover with a phone attached, ready to press Scan QR. Returns the pieces + * the tests below drive; each one ends the handoff itself. + */ +async function scannableHandoff(pngScreenshot: Buffer = QR_PAGE_PNG): Promise<{ + human: Awaited> + events: HandoffEvent[] + handoff: ReturnType +}> { + const port = await startRelayProcess() + const human = await connectHuman(port) + const cdp = fakeCdp() + const events: HandoffEvent[] = [] + const handoff = runHandoff({ + page: fakePage(cdp.cdp, 0, 0, pngScreenshot), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + reason: "The site wants this code scanned with a phone", + logger: noopLogger, + onEvent: (event) => events.push(event), + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "qr-handoff", + relayColdStartMs: 12, + logger: noopLogger, + }) + await until("the phone to see the reason", () => + human.inbox.some((message) => message.type === "state"), + ) + return { human, events, handoff } +} + +/** Every `links` message the phone has been sent so far. */ +function linksSeen( + inbox: RelayMessage[], +): Extract[] { + return inbox.filter( + (message): message is Extract => + message.type === "links", + ) +} + +test("a scan reads the page and sends the human the link it carries", async () => { + const { human, events, handoff } = await scannableHandoff() + + human.send({ type: "scanqr" }) + await until( + "the phone to be sent links", + () => linksSeen(human.inbox).length === 1, + ) + + const answer = linksSeen(human.inbox)[0] + expect(answer?.source).toBe("qr") + expect(answer?.links).toEqual([{ text: QR_PAGE_LINK, kind: "url" }]) + + human.send({ type: "handback" }) + await handoff + expect(events[0]?.qrScans).toBe(1) + expect(events[0]?.qrHits).toBe(1) +}) + +test("a page with no code answers nothing found, and still counts as a scan", async () => { + const { human, events, handoff } = await scannableHandoff(BLANK_PNG) + + human.send({ type: "scanqr" }) + await until( + "the phone to be sent links", + () => linksSeen(human.inbox).length === 1, + ) + expect(linksSeen(human.inbox)[0]?.links).toEqual([]) + + human.send({ type: "handback" }) + await handoff + // A scan that found nothing still happened, and the gap between these two is + // the number worth watching. + expect(events[0]?.qrScans).toBe(1) + expect(events[0]?.qrHits).toBe(0) +}) + +test("a second scan inside the rate limit is dropped, not queued", async () => { + const { human, events, handoff } = await scannableHandoff() + + // A held button, a double tap, or a second holder of the handoff link. The + // limit is enforced here and not on the phone, because the socket behind + // that link is reachable from any HTTP client. + human.send({ type: "scanqr" }) + human.send({ type: "scanqr" }) + human.send({ type: "scanqr" }) + await until( + "the phone to be sent links", + () => linksSeen(human.inbox).length >= 1, + ) + // Long enough for a queued scan to have answered, and well inside the 2s floor. + await Bun.sleep(400) + + expect(linksSeen(human.inbox)).toHaveLength(1) + human.send({ type: "handback" }) + await handoff + expect(events[0]?.qrScans).toBe(1) + // One screenshot for the scan and no other: a takeover casts, it does not + // screenshot, so this is the whole count. + expect(screenshots).toBe(1) +}) + +test("a scan that lands after the handoff ended is not counted or sent", async () => { + // A screenshot is a CDP round trip and the human can hand back while it is + // in flight. What comes back cannot be delivered — the phone has already + // been told it is over — so it is not a scan that happened, and the wide + // event must not claim one. The event is built after the scan task is + // awaited, so the counters it reads are final. + const port = await startRelayProcess() + const human = await connectHuman(port) + const cdp = fakeCdp() + const events: HandoffEvent[] = [] + const handoff = runHandoff({ + page: fakePage(cdp.cdp, 400, 0, QR_PAGE_PNG), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + reason: "The site wants this code scanned with a phone", + logger: noopLogger, + onEvent: (event) => events.push(event), + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "qr-settle-race", + relayColdStartMs: 12, + logger: noopLogger, + }) + await until("the phone to see the reason", () => + human.inbox.some((message) => message.type === "state"), + ) + + human.send({ type: "scanqr" }) + // Inside the 400 ms the fake page holds the screenshot for. + await Bun.sleep(120) + human.send({ type: "handback" }) + + const end = await handoff + expect(end.outcome).toBe("resolved") + expect(linksSeen(human.inbox)).toHaveLength(0) + expect(events[0]?.qrScans).toBe(0) + expect(events[0]?.qrHits).toBe(0) +}) + +test("an approval never scans, whatever the phone sends", async () => { + const port = await startRelayProcess("approval") + const human = await connectHuman(port) + const cdp = fakeCdp() + const events: HandoffEvent[] = [] + const handoff = runHandoff({ + page: fakePage(cdp.cdp), + agentWsUrl: `ws://127.0.0.1:${port}/ws?role=agent`, + options: { + mode: "approval", + reason: "The agent may not move money without a human", + action: "Transfer EUR 12,430.00 to Acme GmbH", + logger: noopLogger, + onEvent: (event) => events.push(event), + }, + timeoutMs: 5000, + url: "https://relay.example/?pt_token=x", + handoffId: "qr-approval", + relayColdStartMs: 12, + logger: noopLogger, + }) + await until("the phone to see the screenshot", () => + human.inbox.some((message) => message.type === "frame"), + ) + + human.send({ type: "scanqr" }) + await Bun.sleep(300) + expect(linksSeen(human.inbox)).toHaveLength(0) + + human.send({ type: "approve" }) + await handoff + expect(events[0]?.qrScans).toBe(0) + // The one screenshot is the approval's own frame; the scan added none. + expect(screenshots).toBe(1) +}) diff --git a/src/core/input.ts b/src/core/input.ts index 14e5881..4a5e915 100644 --- a/src/core/input.ts +++ b/src/core/input.ts @@ -298,9 +298,12 @@ export function createInputTarget(cdp: CdpChannel): InputTarget { appliedCount += 1 return } + case "scanqr": case "handback": case "abort": - // Lifecycle, not input. raiseHand handles these. + // Not input: nothing about these touches the page through this module. + // raiseHand handles them — the first by taking a screenshot, the other + // two by ending the handoff. return } } diff --git a/src/core/png.ts b/src/core/png.ts new file mode 100644 index 0000000..7dcfe96 --- /dev/null +++ b/src/core/png.ts @@ -0,0 +1,288 @@ +/** + * Just enough PNG to hand a screenshot to a QR decoder. + * + * `jsQR` wants what a canvas gives a browser: four bytes per pixel, row-major, + * RGBA. Node has no canvas and no image decoder — but it does have zlib, and a + * PNG is a zlib stream of filtered scanlines. That is the whole of this file: + * about two hundred lines instead of a native dependency that has to build on + * every platform a handraise user runs an agent on. + * + * It decodes the subset that is actually produced here, and refuses the rest + * loudly rather than guessing: + * + * - 8 bits per channel, non-interlaced, deflate, filter method 0. Chromium's + * `Page.captureScreenshot` emits colour type 2 (RGB) for a screenshot and 6 + * (RGBA) when the page has transparency; measured, docs/measurements/05-qr.md. + * - No palette (colour type 3) and no 16-bit depth. Nothing in this repo + * produces either, and a decoder path with no input is a decoder path + * nobody has ever run. + * + * **What it does not check: the CRCs.** Every chunk carries one and this reads + * none of them, because the contract is "Chromium's own screenshot, over an + * in-process CDP call" — there is no lossy channel between the encoder and + * here for a CRC to catch. What it does check is every length and boundary, + * which is the part an attacker controls: a file whose chunk lengths walk off + * the end, whose image data decompresses to the wrong size, or whose header + * claims a size nothing could have produced is rejected by name rather than + * quietly turned into pixels for a security-sensitive classifier. + */ +import { inflateSync } from "node:zlib" + +const SIGNATURE = Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]) + +/** Bytes per pixel by PNG colour type: grey, RGB, grey+alpha, RGBA. */ +const CHANNELS = new Map([ + [0, 1], + [2, 3], + [4, 2], + [6, 4], +]) + +/** Length + type + CRC around every chunk's payload. */ +const CHUNK_OVERHEAD = 12 + +/** IHDR's payload is exactly this long, always. */ +const IHDR_LENGTH = 13 + +/** + * The largest image this will decode. + * + * 8192 on a side, 24 megapixels in total. Both are checked before a byte is + * decompressed, because the header is the only part of a PNG that is cheap to + * believe and every allocation below is sized from it. + * + * The pixel cap is set by what the decoder can finish inside its own deadline, + * not by what a screen could hold: 24 MP measured 1.2 s and the deadline is 6 s + * (docs/measurements/05-qr.md §6). **A 4K viewport at device scale 2 is + * 7680x4320 — 33.2 MP — and is refused**, which is a real limit and not an + * oversight: it is four times the pixels of the 4K screenshot the scan is built + * for, and no page draws a QR code that needs them. A caller who hits this + * should scan at device scale 1. + */ +export const MAX_DIMENSION = 8192 +export const MAX_PIXELS = 24_000_000 + +/** + * The largest compressed image data this will inflate. + * + * A screenshot of a page is a few hundred kilobytes; 32 MB is two orders of + * magnitude of headroom and still a bound. It exists so that a stream is + * refused before `inflateSync` is asked to look at it, rather than after. + */ +export const MAX_COMPRESSED_BYTES = 32 * 1024 * 1024 + +/** A decoded image in the one layout `jsQR` and `ImageData` agree on. */ +export interface RgbaImage { + data: Uint8ClampedArray + width: number + height: number +} + +interface PngHeader { + width: number + height: number + channels: number + /** Bytes the image data must decompress to: a filter byte plus a row, per row. */ + inflatedLength: number +} + +function fail(what: string): never { + throw new Error(`handraise: ${what}`) +} + +/** IHDR is mandatory and always the first chunk, so it is read by position. */ +function readHeader(bytes: Buffer): PngHeader { + if (bytes.length < 8 + CHUNK_OVERHEAD + IHDR_LENGTH) { + fail("the screenshot is not a PNG") + } + if (!bytes.subarray(0, 8).equals(SIGNATURE)) { + fail("the screenshot is not a PNG") + } + if ( + bytes.readUInt32BE(8) !== IHDR_LENGTH || + bytes.subarray(12, 16).toString("ascii") !== "IHDR" + ) { + fail("the PNG does not start with a 13-byte IHDR") + } + const depth = bytes[24] + const colourType = bytes[25] + const compression = bytes[26] + const filterMethod = bytes[27] + const interlace = bytes[28] + const channels = + colourType === undefined ? undefined : CHANNELS.get(colourType) + if ( + depth !== 8 || + interlace !== 0 || + compression !== 0 || + filterMethod !== 0 || + channels === undefined + ) { + fail( + `unsupported PNG (colour type ${colourType}, ${depth} bits, interlace ${interlace}, compression ${compression}, filter method ${filterMethod})`, + ) + } + const width = bytes.readUInt32BE(16) + const height = bytes.readUInt32BE(20) + if ( + width < 1 || + height < 1 || + width > MAX_DIMENSION || + height > MAX_DIMENSION || + width * height > MAX_PIXELS + ) { + fail( + `the PNG claims ${width}x${height}, past the ${MAX_DIMENSION}px / ${MAX_PIXELS}-pixel cap`, + ) + } + return { + width, + height, + channels, + inflatedLength: (width * channels + 1) * height, + } +} + +/** + * The image data, which a PNG may split over any number of IDAT chunks. + * + * The walk is the boundary check: every chunk's declared length has to fit in + * what is left of the file, or the file is lying about its own shape. IEND is + * required, so a truncated stream is a refusal rather than whatever the last + * complete chunk happened to hold. + */ +function collectImageData(bytes: Buffer): Buffer { + const parts: Buffer[] = [] + let total = 0 + let offset = 8 + let ended = false + while (offset + CHUNK_OVERHEAD <= bytes.length) { + const length = bytes.readUInt32BE(offset) + const type = bytes.subarray(offset + 4, offset + 8).toString("ascii") + if (offset + CHUNK_OVERHEAD + length > bytes.length) { + fail(`the PNG's ${type} chunk runs past the end of the file`) + } + if (type === "IEND") { + ended = true + break + } + if (type === "IDAT") { + total += length + if (total > MAX_COMPRESSED_BYTES) { + fail( + `the PNG carries more than ${MAX_COMPRESSED_BYTES} compressed bytes`, + ) + } + parts.push(bytes.subarray(offset + 8, offset + 8 + length)) + } + offset += length + CHUNK_OVERHEAD + } + if (!ended) fail("the PNG has no IEND chunk") + if (parts.length === 0) fail("the PNG carries no image data") + return Buffer.concat(parts) +} + +/** PNG's own predictor, from the spec's Filter type 4. */ +function paeth(left: number, above: number, corner: number): number { + const estimate = left + above - corner + const dLeft = Math.abs(estimate - left) + const dAbove = Math.abs(estimate - above) + const dCorner = Math.abs(estimate - corner) + if (dLeft <= dAbove && dLeft <= dCorner) return left + return dAbove <= dCorner ? above : corner +} + +/** + * Undo one scanline's filter, in place. + * + * Every filter is a difference against the byte to the left, the byte above, + * or both, so a row can only be reconstructed after the row above it. Writing + * into a `Uint8Array` is what makes the arithmetic wrap at 256 the way the + * spec's modulo does. + */ +function unfilterRow( + row: Uint8Array, + above: Uint8Array, + filter: number, + bpp: number, +): void { + for (let i = 0; i < row.length; i++) { + const value = row[i] ?? 0 + const left = i >= bpp ? (row[i - bpp] ?? 0) : 0 + const up = above[i] ?? 0 + const corner = i >= bpp ? (above[i - bpp] ?? 0) : 0 + if (filter === 1) row[i] = value + left + else if (filter === 2) row[i] = value + up + else if (filter === 3) row[i] = value + ((left + up) >> 1) + else if (filter === 4) row[i] = value + paeth(left, up, corner) + } +} + +/** Filtered scanlines (one filter byte each) to raw samples. */ +function unfilter(raw: Buffer, header: PngHeader): Uint8Array { + const stride = header.width * header.channels + const pixels = new Uint8Array(stride * header.height) + const firstAbove = new Uint8Array(stride) + let offset = 0 + for (let y = 0; y < header.height; y++) { + const filter = raw[offset] + offset += 1 + if (filter === undefined || filter > 4) fail(`unknown PNG filter ${filter}`) + const row = pixels.subarray(y * stride, (y + 1) * stride) + row.set(raw.subarray(offset, offset + stride)) + offset += stride + const above = + y === 0 ? firstAbove : pixels.subarray((y - 1) * stride, y * stride) + unfilterRow(row, above, filter, header.channels) + } + return pixels +} + +/** Widen whatever channels the file has to the RGBA a decoder expects. */ +function toRgba(pixels: Uint8Array, header: PngHeader): Uint8ClampedArray { + if (header.channels === 4) return new Uint8ClampedArray(pixels) + const count = header.width * header.height + const rgba = new Uint8ClampedArray(count * 4) + const grey = header.channels < 3 + for (let i = 0; i < count; i++) { + const from = i * header.channels + const to = i * 4 + const first = pixels[from] ?? 0 + rgba[to] = first + rgba[to + 1] = grey ? first : (pixels[from + 1] ?? 0) + rgba[to + 2] = grey ? first : (pixels[from + 2] ?? 0) + rgba[to + 3] = header.channels === 2 ? (pixels[from + 1] ?? 255) : 255 + } + return rgba +} + +/** + * Decode a PNG to RGBA. Throws with a readable message on anything else. + * + * The inflate is bounded by the header, and that is the whole point of doing + * it in that order. A PNG's IHDR says exactly how many bytes its image data + * decompresses to — one filter byte plus one scanline per row — so anything + * else is not a picture: a stream that inflates past it is a zip bomb, and one + * that stops short is a truncated or forged file. Measured without the bound: + * 815 KB of IDAT claiming an 8x8 image allocated 873 MB and then decoded + * happily, because the first 200 bytes were a valid 8x8 image. + * + * `scanPageForLinks` only ever feeds this Chromium's own screenshot, but + * `scanQrLinks` is exported and documented as taking a PNG from anywhere. + */ +export function decodePng(bytes: Buffer): RgbaImage { + const header = readHeader(bytes) + const raw = inflateSync(collectImageData(bytes), { + maxOutputLength: header.inflatedLength, + }) + if (raw.length !== header.inflatedLength) { + fail( + `the PNG's image data decompressed to ${raw.length} bytes, not the ${header.inflatedLength} its header claims`, + ) + } + return { + data: toRgba(unfilter(raw, header), header), + width: header.width, + height: header.height, + } +} diff --git a/src/core/qr-fixtures.ts b/src/core/qr-fixtures.ts new file mode 100644 index 0000000..ae4f10a --- /dev/null +++ b/src/core/qr-fixtures.ts @@ -0,0 +1,48 @@ +/** + * Payloads that must never become an "Open" button, shared by the two tests + * that check the two locks. + * + * The agent classifies a scanned code and the phone checks it again before it + * builds an anchor, because `kind` crosses a socket anybody holding the handoff + * URL can write to. Two locks are only two locks if they check the same thing, + * and the first version of this feature shipped a page that checked the scheme + * and nothing else — so a forged `kind: "url"` on a payload with a tab in it + * got an anchor. One list, asserted in `qr-scan.test.ts` against + * `classifyLink` and in `e2e/ui.spec.ts` against the real page in a real + * browser, with every entry forged as `kind: "url"`. + * + * Not test-only by accident: it lives beside the code it constrains so that a + * new scheme trick is added in one place. + */ +export const NEVER_OPENABLE: readonly string[] = [ + // Schemes. The first two are the ones everybody remembers; the rest are the + // reason this is an allowlist. + "javascript:alert(document.cookie)", + "data:text/html;base64,PHNjcmlwdD5hbGVydCgxKTwvc2NyaXB0Pg==", + "blob:https://example.com/6e0f2b3c-1111-2222-3333-444455556666", + "content://com.android.contacts/contacts/1", + "file:///etc/passwd", + "intent://scan/#Intent;scheme=zxing;end", + "vbscript:msgbox(1)", + // A scheme nested inside an accepted one: only the outer protocol is what + // `new URL` reports, so this has to be refused on the outer one. + "intent://open#Intent;S.url=https%3A%2F%2Fexample.com;end", + // The URL parser deletes the tab and the newline, and the host it ends up + // with is evil.example — while the string on the card still reads good. + "https://good.example\t@evil.example/", + "https://good.example\n@evil.example/", + // No whitespace needed: everything before the `@` is a username. + "https://good.example@evil.example/", + "https://user:secret@evil.example/", + // Invisible by design. A right-to-left override reverses the visible tail of + // a path; a zero-width space hides inside a hostname; the isolates do both + // without leaving a mark on the screen. + "https://example.com/‮gnp.exe", + "https://exam​ple.com/x", + "https://example.com/⁦a⁩b", + // Actions rather than pages: a dialer control sequence and an authenticator + // enrolment. Decoded, shown and copyable — never one tap away. + "tel:*21*1234567890%23", + "tel:+4915112345678", + "otpauth://totp/Example:ada?secret=JBSWY3DPEHPK3PXP&issuer=Example", +] diff --git a/src/core/qr-scan.test.ts b/src/core/qr-scan.test.ts new file mode 100644 index 0000000..2eaa502 --- /dev/null +++ b/src/core/qr-scan.test.ts @@ -0,0 +1,343 @@ +/** + * The decoder, against images rather than mocks. + * + * bun test src/core/qr-scan.test.ts + * + * Two kinds of input, on purpose. `src/core/fixtures/qr-page.png` is a real + * 1280x800 screenshot taken by a real Solari cloud browser — colour type 2 + * (RGB) with the whole range of PNG scanline filters in it, which is the shape + * production actually feeds this code, and which nothing synthetic reproduces. + * Regenerate it with `bun --env-file=.env scripts/measure-qr-decode.ts`. The + * rest are generated here from the `qrcode` dev dependency, so the small, dense, + * rotated, absent and doubled cases cost no binary in the repository. + */ +import { expect, test } from "bun:test" +import { readFileSync } from "node:fs" +import { fileURLToPath } from "node:url" +import { deflateSync } from "node:zlib" +import jsQR from "jsqr" +import QRCode from "qrcode" + +import { decodePng, type RgbaImage } from "./png" +import { NEVER_OPENABLE } from "./qr-fixtures" +import { + classifyLink, + createQrScanner, + MAX_LINK_CHARS, + OPENABLE_SCHEMES, + scanImage, + scanQrLinks, +} from "./qr-scan" + +const FIXTURE = fileURLToPath( + new URL("./fixtures/qr-page.png", import.meta.url), +) + +/** The screenshot that failed the first live e2e run; see the test that uses it. */ +const CENTRED_FIXTURE = fileURLToPath( + new URL("./fixtures/qr-centred.png", import.meta.url), +) + +/** The payload the fixture's page was drawn with; see the measurement script. */ +const FIXTURE_PAYLOAD = `https://verify.example.com/device?token=${"a1b2c3d4".repeat(20)}` + +async function qrImage(text: string, scale = 6): Promise { + return decodePng(await QRCode.toBuffer(text, { scale, margin: 2 })) +} + +/** A white canvas, the quiet zone every QR code needs around it. */ +function blank(width: number, height: number): RgbaImage { + const data = new Uint8ClampedArray(width * height * 4) + data.fill(255) + return { data, width, height } +} + +function paste(into: RgbaImage, image: RgbaImage, x: number, y: number): void { + for (let row = 0; row < image.height; row++) { + const from = row * image.width * 4 + into.data.set( + image.data.subarray(from, from + image.width * 4), + ((y + row) * into.width + x) * 4, + ) + } +} + +/** Quarter turn clockwise, exactly — no resampling, so nothing else changes. */ +function rotate90(image: RgbaImage): RgbaImage { + const turned = blank(image.height, image.width) + for (let y = 0; y < image.height; y++) { + for (let x = 0; x < image.width; x++) { + const from = (y * image.width + x) * 4 + const to = (x * turned.width + (image.height - 1 - y)) * 4 + turned.data.set(image.data.subarray(from, from + 4), to) + } + } + return turned +} + +// --- the images ----------------------------------------------------------- + +test("a real cloud-browser screenshot decodes to the link it carries", () => { + const links = scanQrLinks(readFileSync(FIXTURE)) + expect(links).toEqual([{ text: FIXTURE_PAYLOAD, kind: "url" }]) +}) + +test("a symbol the page resampled decodes, though the plain pass cannot", () => { + // This screenshot failed the first live run of the e2e, and it is kept + // exactly as it came off the browser. The code in it is large, centred and + // perfectly sharp to the eye — but the page drew a 534px image at 420 CSS + // px, so a module is 4.9 pixels wide and `jsQR`'s fixed 8x8 binarizer blocks + // straddle the module boundaries. The plain pass finds nothing; a tight crop + // of the very same pixels decodes. See docs/measurements/05-qr.md. + const shot = readFileSync(CENTRED_FIXTURE) + const image = decodePng(shot) + expect(jsQR(image.data, image.width, image.height)).toBeNull() + + const links = scanQrLinks(shot) + expect(links).toHaveLength(1) + expect(links[0]?.kind).toBe("url") + expect(links[0]?.text).toContain("/verified?pt_token=") +}) + +test("a small symbol decodes", async () => { + expect(scanImage(await qrImage("https://example.com/a", 3))).toEqual([ + "https://example.com/a", + ]) +}) + +test("a dense symbol decodes", async () => { + // 900 characters is version 27 or so: far past anything a login page draws, + // and the case where a wrong scanline filter shows up as garbage rather than + // as a failure to find the code at all. + const long = `https://example.com/?q=${"x".repeat(900)}` + expect(scanImage(await qrImage(long, 5))).toEqual([long]) +}) + +test("a symbol turned on its side decodes", async () => { + const upright = await qrImage("https://example.com/rotated", 6) + expect(scanImage(rotate90(upright))).toEqual(["https://example.com/rotated"]) +}) + +test("a page with no code decodes to nothing", () => { + expect(scanImage(blank(600, 400))).toEqual([]) +}) + +test("two codes on one screen are both reported", async () => { + // Six finder patterns in one image defeat jsQR's locator outright: the + // whole-page pass below finds neither of these. What rescues it is the tiled + // second look, and this is the test that holds that fallback in place — see + // docs/measurements/05-qr.md and the comment on `readTiles`. + const first = await qrImage("https://example.com/first", 9) + const second = await qrImage("https://example.com/second", 9) + const page = blank(1280, 800) + paste(page, first, 80, 200) + paste(page, second, 700, 200) + + const found = scanImage(page) + expect(found.sort()).toEqual([ + "https://example.com/first", + "https://example.com/second", + ]) +}) + +test("the same code is not reported twice", async () => { + // Masking a symbol that has been read is what stops the second pass from + // finding the first one again and calling it a second code. + expect(scanImage(await qrImage("https://example.com/once", 6))).toEqual([ + "https://example.com/once", + ]) +}) + +// --- what the phone may open ---------------------------------------------- + +test("only the three web schemes are openable", () => { + expect([...OPENABLE_SCHEMES].sort()).toEqual(["http:", "https:", "mailto:"]) +}) + +test("a link in an openable scheme is a url", () => { + for (const text of [ + "https://example.com/verify?token=abc", + "http://192.168.0.4:8080/pair", + "mailto:help@example.com", + ]) { + expect(classifyLink(text)).toEqual({ text, kind: "url" }) + } +}) + +test("a dialer string and an authenticator secret are shown, never opened", () => { + // Both are one tap, both are hard to take back, and both came off a page + // nobody vetted: `tel:` can carry a USSD control sequence and `otpauth:` + // enrols a secret in the human's authenticator. They stay decoded and + // copyable; the phone labels them and the human types them where they go. + for (const text of [ + "tel:+4915112345678", + "tel:*21*1234567890%23", + "otpauth://totp/Example:ada?secret=JBSWY3DPEHPK3PXP&issuer=Example", + ]) { + expect(classifyLink(text)).toEqual({ text, kind: "text" }) + } +}) + +test("everything else is text, and stays readable as text", () => { + for (const text of [ + "javascript:alert(document.cookie)", + "JavaScript:alert(1)", + "data:text/html;base64,PHNjcmlwdD5hbGVydCgxKTwvc2NyaXB0Pg==", + "file:///etc/passwd", + "intent://scan/#Intent;scheme=zxing;end", + "vbscript:msgbox(1)", + "WIFI:S:GuestNet;T:WPA;P:hunter2;;", + "BEGIN:VCARD\nVERSION:3.0\nFN:Ada\nEND:VCARD", + "just a sentence someone printed on a poster", + "", + ]) { + expect(classifyLink(text).kind).toBe("text") + } +}) + +test("a link that reads as one host and goes to another is not a url", () => { + for (const text of NEVER_OPENABLE) { + expect(classifyLink(text).kind).toBe("text") + } +}) + +test("a scheme hidden behind whitespace is not a url", () => { + // The URL parser drops tabs and newlines, so it would report `https:` for a + // string whose visible first line says something else entirely. What the + // parser validated and what the phone would show have to be one string. + expect(classifyLink("https://exa\tmple.com/x").kind).toBe("text") + expect(classifyLink("https://example.com/x evil").kind).toBe("text") +}) + +test("surrounding whitespace is trimmed rather than making a link untouchable", () => { + expect(classifyLink(" https://example.com/x\n")).toEqual({ + text: "https://example.com/x", + kind: "url", + }) +}) + +test("an absurdly long payload is capped and demoted to text", () => { + const huge = `https://example.com/?q=${"y".repeat(4000)}` + const link = classifyLink(huge) + expect(link.text).toHaveLength(MAX_LINK_CHARS) + expect(link.text).toBe(huge.slice(0, MAX_LINK_CHARS)) + // Still a valid URL after the cut, and deliberately still openable: the cap + // is about what a phone can render, not about what a scheme may do. + expect(link.kind).toBe("url") +}) + +// --- the PNG decoder ------------------------------------------------------ + +test("a PNG the decoder cannot read says so instead of guessing", async () => { + // Colour type 3 (palette). Nothing here produces one, and a decoder path + // with no input is a path nobody has run — so it is refused by name. + const palette = await QRCode.toBuffer("https://example.com", { + scale: 4, + margin: 2, + // SAFETY: `qrcode` writes a palette PNG when told to use two colours; the + // option is not in its published types, and this test exists to prove the + // decoder rejects exactly that file. + type: "png", + }) + // The generated file is RGBA, so build the refusal case by hand instead: + // take its header and claim colour type 3. + const forged = Buffer.from(palette) + forged[25] = 3 + expect(() => decodePng(forged)).toThrow(/unsupported PNG/) +}) + +test("something that is not a PNG at all says so", () => { + expect(() => decodePng(Buffer.from("not a png, just some bytes"))).toThrow( + /not a PNG/, + ) +}) + +/** A PNG with a chosen IHDR and a chosen (already deflated) IDAT payload. */ +function forgePng(width: number, height: number, idat: Buffer): Buffer { + const chunk = (type: string, body: Buffer): Buffer => { + const head = Buffer.alloc(8) + head.writeUInt32BE(body.length, 0) + head.write(type, 4, "ascii") + // The CRC is never checked by this decoder, so zero is honest filler. + return Buffer.concat([head, body, Buffer.alloc(4)]) + } + const ihdr = Buffer.alloc(13) + ihdr.writeUInt32BE(width, 0) + ihdr.writeUInt32BE(height, 4) + ihdr[8] = 8 + ihdr[9] = 2 + return Buffer.concat([ + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), + chunk("IHDR", ihdr), + chunk("IDAT", idat), + chunk("IEND", Buffer.alloc(0)), + ]) +} + +test("a zip bomb is refused at the size its own header promised", () => { + // A tiny image whose IDAT decompresses to 64 MB. Without a bound the inflate + // allocates all of it and then decodes the first 200 bytes as a valid 8x8 + // picture — 815 KB in, 873 MB of RSS, and no error at all. The header says + // how many bytes a PNG's image data comes to, so the inflate is capped by it. + const bomb = forgePng(8, 8, deflateSync(Buffer.alloc(64 * 1024 * 1024))) + expect(bomb.length).toBeLessThan(200_000) + + const before = process.memoryUsage().rss + expect(() => decodePng(bomb)).toThrow() + const grew = (process.memoryUsage().rss - before) / (1024 * 1024) + // The bomb is 64 MB uncompressed; refusing it must not cost 64 MB. + expect(grew).toBeLessThan(32) +}) + +test("the pixel cap is the one the decoder can finish inside its deadline", () => { + // The cap is not "what a screen could hold" — it is what the decode finishes + // inside `DECODE_TIMEOUT_MS` with margin, and the two are documented against + // each other. A header at the cap gets past `readHeader` and fails later, on + // its (deliberately absent) image data; one pixel over is refused by the cap + // itself. The two different messages are what prove which check fired. + const tiny = deflateSync(Buffer.alloc(64)) + expect(() => decodePng(forgePng(6000, 4000, tiny))).toThrow(/decompressed to/) + expect(() => decodePng(forgePng(6000, 4001, tiny))).toThrow(/past the .* cap/) + + // And the case the comment now names out loud: a 4K viewport at device scale + // 2 is 7680x4320, which is over the cap and refused. + expect(() => decodePng(forgePng(7680, 4320, tiny))).toThrow(/past the .* cap/) +}) + +test("a header that claims more pixels than any screen is refused", () => { + // 65535x65535 is four billion pixels, and every allocation in the decoder is + // sized from these two numbers. Refused before a byte is inflated. + expect(() => + decodePng(forgePng(65_535, 65_535, deflateSync(Buffer.alloc(64)))), + ).toThrow(/past the .* cap/) +}) + +// --- the worker ------------------------------------------------------------ + +test("a decode that fails does not poison the scanner", async () => { + // The worker answers a refusal with a message rather than dying, so the + // thread survives a hostile PNG and the next scan does not pay a startup — + // or, worse, post into a handle that is dead and wait out the whole deadline + // finding out. + const scanner = createQrScanner() + try { + await expect( + scanner.scan(Buffer.from("not a png, just some bytes")), + ).rejects.toThrow(/not a PNG/) + + const links = await scanner.scan(readFileSync(FIXTURE)) + expect(links).toEqual([{ text: FIXTURE_PAYLOAD, kind: "url" }]) + } finally { + await scanner.close() + } +}) + +test("a closed scanner refuses instead of starting another thread", async () => { + const scanner = createQrScanner() + await scanner.close() + await expect(scanner.scan(readFileSync(FIXTURE))).rejects.toThrow( + /scanner closed/, + ) + // Idempotent, and safe from a `finally` that runs twice. + await scanner.close() +}) diff --git a/src/core/qr-scan.ts b/src/core/qr-scan.ts new file mode 100644 index 0000000..8aabacb --- /dev/null +++ b/src/core/qr-scan.ts @@ -0,0 +1,459 @@ +/** + * Reading the QR codes off the page the agent is stuck on. + * + * The case this exists for is a device-change check: reCAPTCHA, a WhatsApp Web + * login, an authenticator enrolment, a payment code. The site draws a QR code + * and says "scan this with your phone" — and the human handraise put in front + * of it *is* on a phone, looking at the code through their own screen. A phone + * cannot scan itself. Today that needs a second device. + * + * So the agent reads the code instead and sends the human the link. + * + * Three decisions, all argued in docs/adr/0008-qr-passthrough.md: + * + * - It decodes in the agent process, from a fresh full-resolution + * `page.screenshot()`. Not from the cast frame, which is scaled to 800px + * and JPEG-compressed until a dense symbol is mush; and not in the remote + * page, whose JavaScript belongs to whoever the agent got stuck on. + * - Only on request. A scan is a screenshot plus a decode, so scanning every + * frame would cost that on a stream that already paces itself to a phone. + * - It classifies, it never opens. The agent process fetches nothing; the + * phone offers an "Open" button, and only for the schemes below. + */ +import { Worker } from "node:worker_threads" +import jsQR, { type QRCode } from "jsqr" +import type { Page } from "playwright-core" +import { decodePng, type RgbaImage } from "./png" +import type { QrWorkerResult } from "./qr-worker" + +/** + * Whether the phone may offer to open this, or only to copy it. + * + * A QR code is an arbitrary string from a page the agent did not choose. Most + * of them are links, and the useful ones are; the rest are a wifi credential, a + * vCard, a plain sentence — worth showing, never worth handing to a browser. + */ +export type LinkKind = "url" | "text" + +export interface ScannedLink { + /** What the code carried, trimmed and capped. Shown as text either way. */ + text: string + kind: LinkKind +} + +/** + * The schemes a phone is offered an "Open" button for. + * + * An allowlist and not a blocklist, because the interesting half of this list + * is the half nobody thinks of: `javascript:` and `data:` are the two everyone + * remembers, and `intent:`, `file:`, `content:`, `blob:` and whatever a phone + * browser ships next year are the ones a blocklist would have let through. + * + * Three, not five. `tel:` and `otpauth:` were on this list and are not any + * more: opening one hands a dialer a string that can be a USSD control + * sequence, and opening the other enrols an attacker-chosen secret in the + * human's authenticator. Both are one tap, both are hard to take back, and + * both come from a page nobody vetted. They are still decoded, still shown in + * full and still copyable — with a label that says what they are — and the + * human types or pastes them into the app that should have them. See + * docs/adr/0008-qr-passthrough.md. + * + * The phone checks this again before it builds the anchor. Two locks on one + * door on purpose: the agent's `kind` travels over a socket the human's link + * can reach, so it is a hint the page must not have to trust. + */ +export const OPENABLE_SCHEMES: ReadonlySet = new Set([ + "http:", + "https:", + "mailto:", +]) + +/** + * A QR code holds up to 4296 characters. Past a couple of thousand it is not a + * link any more, and the phone has to render it as one line of text. + */ +export const MAX_LINK_CHARS = 2048 + +/** + * How many codes one scan reports. + * + * Pages that show a QR code show one. Two is what makes "there is more than + * one here" sayable instead of silently picking; past that the sheet is a list + * nobody reads on a phone, and the human can scroll the page and scan again. + */ +export const MAX_CODES = 2 + +/** Cap on the screenshot itself: a page that cannot paint must not hang a scan. */ +const SCREENSHOT_TIMEOUT_MS = 5_000 + +/** + * Characters that make a link read as something it is not. + * + * Two families, one rule. Whitespace and the C0/C1 controls, because the URL + * parser silently deletes a tab or a newline and what it validated is then a + * different string from what a human reads. And the Unicode formatting + * controls, because they are invisible by design: a right-to-left override + * reverses the visible tail of a path, and a zero-width space hides inside a + * hostname. A link never needs any of them. + */ +function hasUnsafeCharacter(text: string): boolean { + for (const character of text) { + const code = character.codePointAt(0) ?? 0 + if (code <= 0x20 || (code >= 0x7f && code <= 0x9f)) return true + // Zero-width and bidi formatting: U+200B-U+200F, U+202A-U+202E, + // U+2066-U+2069. + if (code >= 0x200b && code <= 0x200f) return true + if (code >= 0x202a && code <= 0x202e) return true + if (code >= 0x2066 && code <= 0x2069) return true + } + return false +} + +function isOpenable(text: string): boolean { + if (text.length === 0 || hasUnsafeCharacter(text)) return false + try { + const url = new URL(text) + // Credentials in the authority are the oldest way to make a link read as + // one host and go to another: everything before the `@` is a username, and + // a phone screen is exactly where that fits in the visible part. No + // device-change link has ever needed them. + if (url.username !== "" || url.password !== "") return false + return OPENABLE_SCHEMES.has(url.protocol) + } catch { + // Not a URL at all: a wifi credential, a vCard, a sentence. + return false + } +} + +/** Decide what one code's payload is, and what the phone may do with it. */ +export function classifyLink(payload: string): ScannedLink { + const text = payload.trim().slice(0, MAX_LINK_CHARS) + return { text, kind: isOpenable(text) ? "url" : "text" } +} + +/** + * Paint over a symbol that has already been read. + * + * `jsQR` returns the first code it finds and has no way to ask for the next + * one, so the only way to know whether the page holds a second is to remove the + * first and look again. White, because that is the quiet zone every QR code is + * already surrounded by. + */ +function maskOut(image: RgbaImage, location: QRCode["location"]): void { + const xs = [ + location.topLeftCorner.x, + location.topRightCorner.x, + location.bottomLeftCorner.x, + location.bottomRightCorner.x, + ] + const ys = [ + location.topLeftCorner.y, + location.topRightCorner.y, + location.bottomLeftCorner.y, + location.bottomRightCorner.y, + ] + const left = Math.max(0, Math.floor(Math.min(...xs)) - 1) + const right = Math.min(image.width - 1, Math.ceil(Math.max(...xs)) + 1) + const top = Math.max(0, Math.floor(Math.min(...ys)) - 1) + const bottom = Math.min(image.height - 1, Math.ceil(Math.max(...ys)) + 1) + for (let y = top; y <= bottom; y++) { + const row = y * image.width + for (let x = left; x <= right; x++) { + image.data.fill(255, (row + x) * 4, (row + x) * 4 + 4) + } + } +} + +/** Read up to `MAX_CODES` payloads, painting each out before looking again. */ +function readRepeatedly(image: RgbaImage): string[] { + const found: string[] = [] + for (let pass = 0; pass < MAX_CODES; pass++) { + const code = jsQR(image.data, image.width, image.height) + if (!code) break + // A payload seen twice is the same code found again, not a second one. + if (code.data.length > 0 && !found.includes(code.data)) + found.push(code.data) + maskOut(image, code.location) + } + return found +} + +/** + * How much bigger the second look is. Two is enough and three costs 4x the + * pixels for nothing (both decode the fixture; measured in docs/measurements/05-qr.md). + */ +const MAGNIFY = 2 + +/** + * The most pixels one retry pass may allocate. + * + * This is the number that sets what a scan costs, not `MAX_PIXELS`: the 2x look + * is four times the source and it is the expensive pass by a wide margin. + * Measured (docs/measurements/05-qr.md §6), a 24 MP image too big to magnify + * decodes in 1.2 s, while a 10 MP one that magnifies to 40 MP takes 2.8 to + * 3.7 s. + * + * 34 MP is exactly what a 4K screenshot needs — 3840x2160 is 8.3 MP and + * magnifies to 33.2 — and nothing more. That input measured 2.1 s against a 6 s + * deadline. Set it higher and a legitimate scan starts being killed by the + * deadline instead of answered; set it lower and a 4K page loses the retry that + * makes a resampled code readable at all. + */ +const MAX_SCAN_PIXELS = 34_000_000 + +/** + * Look again, twice the size. + * + * Nearest-neighbour, so not one new pixel of information — and that is the + * point. `jsQR` binarizes in fixed 8x8 blocks, and a page that draws its code + * at a size the browser has to resample lands a module boundary in the middle + * of a block. A symbol that is perfectly sharp to the eye then fails to be + * *located* while a tight crop of the same pixels decodes: the failure is the + * block grid, not the image. Doubling it puts about ten pixels under each + * module and the blocks line up again. + * + * Found the hard way. `src/core/fixtures/qr-centred.png` is the screenshot that + * failed the first live run of the e2e, kept exactly as it came off the browser. + */ +function magnify(image: RgbaImage, factor: number): RgbaImage { + const width = image.width * factor + const height = image.height * factor + const data = new Uint8ClampedArray(width * height * 4) + for (let y = 0; y < height; y++) { + const row = Math.floor(y / factor) * image.width + for (let x = 0; x < width; x++) { + const from = (row + Math.floor(x / factor)) * 4 + data.set(image.data.subarray(from, from + 4), (y * width + x) * 4) + } + } + return { data, width, height } +} + +/** Each tile's share of a dimension. Four of them, anchored at the corners. */ +const TILE_SHARE = 0.6 + +function cropTile( + image: RgbaImage, + x: number, + y: number, + width: number, + height: number, +): RgbaImage { + const data = new Uint8ClampedArray(width * height * 4) + for (let row = 0; row < height; row++) { + const from = ((y + row) * image.width + x) * 4 + data.set(image.data.subarray(from, from + width * 4), row * width * 4) + } + return { data, width, height } +} + +/** + * The fallback, and it is not a nicety. + * + * `jsQR` locates a symbol by its three finder patterns, and two symbols on one + * screen put six of them in front of it: on a 1280x800 page with two 260px + * codes it finds *neither*, where each quarter on its own decodes cleanly + * (docs/measurements/05-qr.md). So when the whole image comes back empty, look + * again at four overlapping corners. It costs a second decode only on the pass + * that already failed, and it is what makes "two codes on the page" a result + * rather than "no QR code found" on a page that visibly has two. + */ +function readTiles(image: RgbaImage): string[] { + const width = Math.floor(image.width * TILE_SHARE) + const height = Math.floor(image.height * TILE_SHARE) + if (width < 1 || height < 1) return [] + const found: string[] = [] + for (const x of [0, image.width - width]) { + for (const y of [0, image.height - height]) { + if (found.length >= MAX_CODES) return found + const tile = cropTile(image, x, y, width, height) + const code = jsQR(tile.data, tile.width, tile.height) + if (code && code.data.length > 0 && !found.includes(code.data)) { + found.push(code.data) + } + } + } + return found +} + +/** + * Every QR payload in an image, in the order they are found. + * + * Three looks, each one earning its place, and the second and third only run + * when the one before found nothing: + * + * 1. the image as it came, which is the answer almost every time; + * 2. the image at 2x, for a symbol the page drew at a resampled size — the + * failure that is invisible to the eye (see `magnify`); + * 3. four overlapping corners, for two codes on one screen, which defeat the + * locator outright (see `readTiles`). + * + * The image is modified in place — each symbol is painted out before the next + * pass — so pass a decode that is not needed afterwards. + */ +export function scanImage(image: RgbaImage): string[] { + const whole = readRepeatedly(image) + if (whole.length > 0) return whole + if (image.width * image.height * MAGNIFY * MAGNIFY <= MAX_SCAN_PIXELS) { + const bigger = readRepeatedly(magnify(image, MAGNIFY)) + if (bigger.length > 0) return bigger + } + return readTiles(image) +} + +/** Read the QR codes in a PNG screenshot and say what the phone may do with each. */ +export function scanQrLinks(screenshot: Buffer): ScannedLink[] { + return scanImage(decodePng(screenshot)).map(classifyLink) +} + +/** + * Take a fresh screenshot of the page and read its QR codes. + * + * A new screenshot rather than the newest cast frame: the cast is scaled to + * 800px wide and encoded at JPEG quality 60, a profile chosen for reading a + * login form and one that destroys a dense symbol's modules. PNG rather than + * JPEG for the same reason — the decoder wants edges, not a small file. + * + * The decode itself goes to `scanner`, which owns a worker thread: the + * screenshot is a CDP round trip and yields, but the decode is pure CPU and + * would otherwise stall the agent's loop for as long as it takes. + */ +export async function scanPageForLinks( + page: Page, + scanner: QrScanner, +): Promise { + const shot = await page.screenshot({ + type: "png", + timeout: SCREENSHOT_TIMEOUT_MS, + }) + return scanner.scan(shot) +} + +/** + * A decoder on a thread of its own, for one handoff. + * + * The worker starts on the first scan and not before — most handoffs never + * scan anything, and a thread nobody uses still costs a megabyte and a + * startup. It is reused for every later scan of the same handoff, and + * `close()` at settle is what keeps it from outliving one. + */ +export interface QrScanner { + /** Decode one PNG. Rejects on a refusal, a worker failure, or the deadline. */ + scan(png: Buffer): Promise + /** Stop the worker. Idempotent, never throws, safe from a `finally`. */ + close(): Promise +} + +/** + * How long one decode may take before the worker is assumed lost. + * + * Derived from `MAX_SCAN_PIXELS` rather than picked: the worst input the caps + * admit is a 4K screenshot magnified to 34 MP, which measured 2.1 s + * (docs/measurements/05-qr.md §6). Six seconds is close to three times that, + * which is the margin an agent host under load needs — and it stays inside the + * phone's own 12 s wait even after a 5 s screenshot, so a human is never told + * "the agent didn't answer" about a scan that is still coming. + * + * It was three seconds, and three was wrong: a 10 MP screenshot measured 3.7 s + * and would have been killed on the way to an answer it already had. Past this, + * terminating is the only lever there is over a worker that has stopped + * answering — which is exactly why the work is over there. + */ +const DECODE_TIMEOUT_MS = 6_000 + +/** + * Where the worker's code is, in a source tree and in a published package. + * + * `import.meta.url` is this file under bun and `dist/index.js` in the bundle, + * and the worker sits beside each of them under its own extension. A consumer + * who re-bundles handraise has to keep `qr-worker.js` next to the entry it is + * resolved from; that is the cost of a worker being a file rather than a + * function. + */ +function workerUrl(): URL { + const here = import.meta.url + const file = here.endsWith(".ts") ? "./qr-worker.ts" : "./qr-worker.js" + return new URL(file, here) +} + +/** Start a decoder for one handoff. The worker itself is lazy. */ +export function createQrScanner(): QrScanner { + let worker: Worker | null = null + let closed = false + + const start = (): Worker => { + const started = new Worker(workerUrl()) + // A pool of one, and nothing else in the process waits on it: an idle + // worker must not be the reason a script does not exit. + started.unref() + // Two jobs, and both outlive any single scan. An `'error'` with no listener + // is a throw out of an EventEmitter, and between scans this worker has no + // other one — so it is never without this. And a worker that has errored is + // dead: forgetting it here is what makes the next scan start a fresh one + // instead of posting into a thread that will never answer and burning the + // whole deadline finding out. + started.on("error", () => { + if (worker === started) worker = null + void started.terminate() + }) + return started + } + + return { + scan(png) { + if (closed) return Promise.reject(new Error("handraise: scanner closed")) + if (!worker) worker = start() + const live = worker + return new Promise((resolve, reject) => { + const detach = (): void => { + clearTimeout(timer) + live.off("message", onMessage) + live.off("error", onError) + live.off("exit", onExit) + } + const onMessage = (result: QrWorkerResult): void => { + detach() + if (result.error) reject(new Error(result.error)) + else resolve(result.links ?? []) + } + const onError = (error: Error): void => { + detach() + reject(error) + } + const onExit = (code: number): void => { + detach() + // A worker that exited took this answer with it, and it will not be + // there for the next scan either. + worker = null + reject(new Error(`handraise: the QR worker exited with ${code}`)) + } + const timer = setTimeout(() => { + detach() + worker = null + void live.terminate() + reject( + new Error( + `handraise: the QR decode took over ${DECODE_TIMEOUT_MS}ms`, + ), + ) + }, DECODE_TIMEOUT_MS) + timer.unref?.() + live.on("message", onMessage) + live.on("error", onError) + live.on("exit", onExit) + // Transferred, not copied: a screenshot is megabytes, and this is the + // one place the whole of it crosses a thread boundary. + const bytes = new Uint8Array(png) + live.postMessage(bytes, [bytes.buffer]) + }) + }, + + async close() { + closed = true + const live = worker + worker = null + if (!live) return + await live.terminate().catch(() => undefined) + }, + } +} diff --git a/src/core/qr-worker.ts b/src/core/qr-worker.ts new file mode 100644 index 0000000..bb537a6 --- /dev/null +++ b/src/core/qr-worker.ts @@ -0,0 +1,41 @@ +/** + * The decode, on a thread that is not the agent's. + * + * A PNG decode plus up to three `jsQR` passes is pure synchronous CPU, and on + * a 3840x2160 screenshot it is seconds of it (docs/measurements/05-qr.md). On + * the agent's own loop that stalls everything handraise promises to keep + * working while a human is looking at the page: the frame pump, the handback + * the human is about to send, the timeout, the browser's disconnect. So the + * work happens here instead, and the loop stays free to answer. + * + * The protocol is one message each way and nothing else. In: the PNG bytes, + * transferred rather than copied. Out: `{ links }` or `{ error }` — never a + * throw, because a worker that dies takes its answer with it and the caller's + * deadline is what has to notice. + * + * This file is a build entry of its own (`dist/qr-worker.js`), because a + * worker is loaded by URL at runtime and cannot be bundled into the caller. + */ +import { parentPort } from "node:worker_threads" + +import { type ScannedLink, scanQrLinks } from "./qr-scan" + +/** What the worker sends back. Exactly one of the two fields is present. */ +export interface QrWorkerResult { + links?: ScannedLink[] + error?: string +} + +if (parentPort) { + const port = parentPort + port.on("message", (png: Uint8Array) => { + try { + port.postMessage({ links: scanQrLinks(Buffer.from(png)) }) + } catch (error) { + // Every refusal the decoder makes is a message, not a crash: a hostile + // PNG must not be able to take the worker down and cost the next scan + // its startup. + port.postMessage({ error: String(error) }) + } + }) +} diff --git a/src/core/raise-hand.ts b/src/core/raise-hand.ts index 9556739..cf74cbb 100644 --- a/src/core/raise-hand.ts +++ b/src/core/raise-hand.ts @@ -39,6 +39,7 @@ import type { import { notifyWebhook } from "../webhook" import { NO_FOCUS, probeFocus } from "./focus" import { createInputTarget } from "./input" +import { createQrScanner, scanPageForLinks } from "./qr-scan" import { DEFAULT_PROFILE, type FramePump, startFramePump } from "./screencast" import { type ApprovalFrame, captureApprovalFrame } from "./snapshot" import { connectRelay, type RelayConnection } from "./socket" @@ -61,6 +62,14 @@ const RELAY_SLACK_MS = 5 * 60_000 */ const STORAGE_STATE_TIMEOUT_MS = 5_000 +/** + * Floor between two QR scans. Two seconds is the shortest gap at which a + * second scan can say something new — the human has to move the page for it + * to — and it is long enough that a stuck button cannot turn a live cast into + * a screenshot loop. + */ +const QR_SCAN_INTERVAL_MS = 2_000 + /** Resolve `promise`, or reject with `label` if it has not settled in `ms`. */ function withTimeout( promise: Promise, @@ -376,6 +385,64 @@ export async function runHandoff(run: HandoffRun): Promise { }) } + /** + * Scans the human asked for and answers they were actually sent. + * + * Both count delivered results, not attempts: a scan whose answer is thrown + * away because the handoff settled while it was in flight is not something + * anybody was told, and the event is built after the counters are frozen so + * it cannot report a number that changed afterwards. + */ + let qrScans = 0 + let qrHits = 0 + // One scan at a time, and never two inside the interval. A scan costs a + // full-resolution screenshot of a page that is already casting to a phone, + // so a held button must not be able to turn into a screenshot loop. The + // phone and the relay enforce the same floor; this is the one that counts, + // because the socket behind the handoff URL is reachable from any HTTP client. + let scanning = false + let lastScanAt = 0 + // The decode runs on a worker thread. The scanner object is free — it starts + // no thread until the first scan — so it is made here and closed at + // teardown unconditionally. `scanTask` is the handle teardown waits on, so a + // decode is never still running against a page the caller has moved on from. + const scanner = createQrScanner() + let scanTask: Promise = Promise.resolve() + + /** + * Read the QR codes on the page and send the human what they carry. + * + * Off the critical path, like the focus probe — nothing on the handoff's own + * path awaits it — but retained, so teardown can. + */ + const scanQr = (): void => { + const now = Date.now() + if (scanning || over || now - lastScanAt < QR_SCAN_INTERVAL_MS) return + lastScanAt = now + scanning = true + scanTask = scanPageForLinks(page, scanner) + .then((links) => { + // Checked here, between the screenshot and the answer: the handoff can + // settle while a CDP round trip is in flight, and an answer nobody can + // be sent is not a scan that happened. + if (over) return undefined + qrScans += 1 + if (links.length > 0) qrHits += 1 + return link?.send({ type: "links", links, source: "qr" }) + }) + // A screenshot fails when the page is closing, which is a handoff that + // is about to end as `disconnected` — the phone gets that, and its own + // deadline releases the button. Answering with an empty list here would + // tell the human the page has no code on it, which is a different and + // untrue thing. + .catch((error) => { + logger.warn("qr_scan_failed", { error: String(error) }) + }) + .finally(() => { + scanning = false + }) + } + const onHuman = (message: HumanToAgent): void => { const ending = endingFor(mode, message.type) if (ending) { @@ -386,6 +453,12 @@ export async function runHandoff(run: HandoffRun): Promise { // abandoned, so no further input may run against it. An approval never // injects anything at all: the human is answering, not driving. if (terminal || mode === "approval") return + // The one message that asks the agent about the page instead of changing + // it, so it is answered before the input path and needs no frame metadata. + if (message.type === "scanqr") { + scanQr() + return + } // Input can only be mapped once a frame has defined the coordinate space. const meta = pump?.lastMeta() if (!meta || !input) return @@ -544,6 +617,12 @@ export async function runHandoff(run: HandoffRun): Promise { clearTimeout(timer) browser?.off("disconnected", onGone) page.off("close", onGone) + // Let a scan that was in flight when the handoff settled finish reporting + // itself, then stop its worker. Awaited before the wide event is built, so + // `qrScans` and `qrHits` are frozen by the time it reads them; the decode + // has its own six-second deadline, so this cannot wait on a hung thread. + await scanTask + await scanner.close() await pump?.stop() // The ending must reach the phone, so wait briefly for a reconnect if the // socket is momentarily down rather than dropping it like a stale frame. @@ -565,6 +644,8 @@ export async function runHandoff(run: HandoffRun): Promise { framesSent, bytesSent, inputsApplied: input?.applied() ?? 0, + qrScans, + qrHits, reconnects: connection.stats().reconnects, storageStateCaptured: storageState !== undefined, } diff --git a/src/core/socket.test.ts b/src/core/socket.test.ts index 34aef59..cc1333d 100644 --- a/src/core/socket.test.ts +++ b/src/core/socket.test.ts @@ -356,6 +356,7 @@ const SAMPLES = { key: { type: "key", key: "Enter" }, clear: { type: "clear" }, scroll: { type: "scroll", fdy: 40 }, + scanqr: { type: "scanqr" }, handback: { type: "handback" }, abort: { type: "abort" }, approve: { type: "approve" }, @@ -364,7 +365,16 @@ const SAMPLES = { /** Which mode's relay routes which of them (`HUMAN_MESSAGES` in guest/server.js). */ const ROUTED_BY = { - takeover: ["tap", "char", "key", "clear", "scroll", "handback", "abort"], + takeover: [ + "tap", + "char", + "key", + "clear", + "scroll", + "scanqr", + "handback", + "abort", + ], approval: ["approve", "deny"], } satisfies Record diff --git a/src/core/socket.ts b/src/core/socket.ts index 3b0ad22..9c75f93 100644 --- a/src/core/socket.ts +++ b/src/core/socket.ts @@ -144,6 +144,7 @@ export function connectRelay(options: RelayConnectionOptions): RelayConnection { case "key": case "clear": case "scroll": + case "scanqr": case "handback": case "abort": case "approve": diff --git a/src/events.ts b/src/events.ts index 724e450..870ebd0 100644 --- a/src/events.ts +++ b/src/events.ts @@ -47,6 +47,17 @@ export interface HandoffEvent { * in approval mode, which injects nothing. */ inputsApplied: number + /** + * QR scans the human asked for and the agent performed. Requests dropped by + * the rate limit are not counted — they cost nothing and happened only in + * the sense that a button was pressed twice. Always 0 in approval mode, + * which offers no scan. + */ + qrScans: number + /** Of those, the ones that found at least one code. `qrScans - qrHits` is + * how often the human was told "nothing here", which is the number worth + * watching: it is either a page that has no code or a decode that failed. */ + qrHits: number /** Agent-socket reconnects during the handoff (the 60 s idle cut, drops). */ reconnects: number /** Whether cookies + localStorage were captured after a handback. */ diff --git a/src/index.ts b/src/index.ts index a87f5ef..88b9cc5 100644 --- a/src/index.ts +++ b/src/index.ts @@ -28,6 +28,14 @@ export type { HandoffChannel, TakeoverChannelHandoff, } from "./channels" +export { + createQrScanner, + type LinkKind, + OPENABLE_SCHEMES, + type QrScanner, + type ScannedLink, + scanQrLinks, +} from "./core/qr-scan" export { raiseHand } from "./core/raise-hand" export { HandraiseError, diff --git a/src/relay/guest-source.ts b/src/relay/guest-source.ts index cb81983..a978734 100644 --- a/src/relay/guest-source.ts +++ b/src/relay/guest-source.ts @@ -43,28 +43,93 @@ const PORT = Number(process.argv[2] || process.env.HANDRAISE_RELAY_PORT || 3000) */ const AGENT_KEY = process.argv[3] || process.env.HANDRAISE_AGENT_KEY || "" +/** + * Every message type on the wire, in both directions, named once. + * + * This file is plain JavaScript, so comparing a message against a bare quoted + * string is unchecked: a typo is not a compile error, it is a message that is + * silently never matched. The literal unions in src/relay/protocol.ts do that + * work for the rest of the codebase; these constants are their counterpart + * here, and \`relay.test.ts\` asserts the two sets against each other so neither + * can grow a member alone. + * + * The mobile page below gets this same object injected at serve time — one + * definition for the relay and the page it serves, never two that can drift. + */ +const MSG = { + // agent -> human + FRAME: "frame", + STATE: "state", + FOCUS: "focus", + LINKS: "links", + ENDED: "ended", + // human -> agent + TAP: "tap", + SCANQR: "scanqr", + CHAR: "char", + KEY: "key", + CLEAR: "clear", + SCROLL: "scroll", + HANDBACK: "handback", + ABORT: "abort", + APPROVE: "approve", + DENY: "deny", + // either direction + PING: "ping", + PONG: "pong", +} + +/** The two things a handoff can ask of a human. */ +const MODE = { TAKEOVER: "takeover", APPROVAL: "approval" } + +/** + * The URL schemes the page may offer an "Open" button for, from a QR code the + * agent read off whatever site it got stuck on. + * + * Three. \`tel:\` and \`otpauth:\` are deliberately not here: opening one hands a + * dialler a string that can be a control sequence, and opening the other + * enrols an attacker-chosen secret in an authenticator. Both are shown and + * copyable with a label that says what they are. + * + * The agent classifies each link before it sends it, and the page checks the + * scheme again against this list. Both locks are needed: the human's link is a + * bearer URL and the socket behind it is reachable from any HTTP client, so + * \`kind: "url"\` is a hint the page must not have to trust. Asserted equal to + * \`OPENABLE_SCHEMES\` in src/core/qr-scan.ts by relay.test.ts. + */ +const OPENABLE_SCHEMES = ["http:", "https:", "mailto:"] + /** * What this handoff asks of the human: \`takeover\` (drive the page) or * \`approval\` (answer one question about one screenshot). It arrives as argv * and never as a message, so no client can talk the relay into the other set. */ -const MODE = - (process.argv[4] || process.env.HANDRAISE_MODE) === "approval" - ? "approval" - : "takeover" +const HANDOFF_MODE = + (process.argv[4] || process.env.HANDRAISE_MODE) === MODE.APPROVAL + ? MODE.APPROVAL + : MODE.TAKEOVER /** The human messages this relay forwards. Everything else from that side is dropped. */ const HUMAN_MESSAGES = new Set( - MODE === "approval" - ? ["approve", "deny"] - : ["tap", "char", "key", "clear", "scroll", "handback", "abort"], + HANDOFF_MODE === MODE.APPROVAL + ? [MSG.APPROVE, MSG.DENY] + : [ + MSG.TAP, + MSG.CHAR, + MSG.KEY, + MSG.CLEAR, + MSG.SCROLL, + MSG.SCANQR, + MSG.HANDBACK, + MSG.ABORT, + ], ) /** * The human messages that end a handoff, in either mode. They are held for an * agent that is not connected at the moment, and they stop the frame replay. */ -const TERMINAL_HUMAN = new Set(["handback", "abort", "approve", "deny"]) +const TERMINAL_HUMAN = new Set([MSG.HANDBACK, MSG.ABORT, MSG.APPROVE, MSG.DENY]) /** Must equal HEARTBEAT_INTERVAL_MS in src/relay/protocol.ts (asserted in relay.test.ts). */ const HEARTBEAT_INTERVAL_MS = 20000 @@ -79,10 +144,33 @@ const OP_PONG = 0xa /** A screencast frame is ~12-65 KB of base64; anything past this is a bug or an attack. */ const MAX_MESSAGE_BYTES = 8 * 1024 * 1024 +/** + * The ceiling for a message from the human, which is a different number + * entirely. + * + * Every message this side may send is a handful of fields — a tap is two + * integers, the longest is one character of text — so four kilobytes is + * already a thousandfold of what any of them needs. The agent's frames are the + * reason the cap above is megabytes; a bearer-link holder padding an accepted + * \`scanqr\` to eight of them is the reason this one is not. Enforced in the + * reader, before anything is parsed. + */ +const MAX_HUMAN_MESSAGE_BYTES = 4 * 1024 + +/** + * The relay's own floor between two scans, in milliseconds. + * + * The core enforces this too and its copy is the one that protects the + * browser. This one protects the relay and the agent's socket: a burst of + * accepted \`scanqr\` objects still costs a forward, a parse on the other side + * and a wake-up each, and none of that is the core's to refuse. + */ +const SCAN_INTERVAL_MS = 2000 + /** Grace before a replaced/closed socket is force-destroyed if it hangs on. */ const CLOSE_GRACE_MS = 1000 -const PONG = JSON.stringify({ type: "pong" }) +const PONG = JSON.stringify({ type: MSG.PONG }) /** role -> peer. At most one connection per role; a new one replaces the old. */ const peers = new Map() @@ -118,6 +206,9 @@ let pendingForAgent = null */ let humanEnded = false +/** When the relay last forwarded a \`scanqr\`. See SCAN_INTERVAL_MS. */ +let lastScanAt = 0 + /** * Forget everything that shows the remote page. Not the ending, which a late * human still has to be told, and not a human answer still waiting for its @@ -154,7 +245,7 @@ function encodeFrame(payload, opcode) { * frames and fragmentation (browsers do not fragment, but proxies may). * \`onMessage(payload, opcode)\` fires once per complete application message. */ -function createReader(onMessage, onPing, onClose) { +function createReader(onMessage, onPing, onClose, maxBytes) { let buffered = Buffer.alloc(0) let fragments = [] let fragmentBytes = 0 @@ -178,7 +269,7 @@ function createReader(onMessage, onPing, onClose) { length = Number(buffered.readBigUInt64BE(2)) offset = 10 } - if (length > MAX_MESSAGE_BYTES) { + if (length > maxBytes) { onClose() return } @@ -212,7 +303,7 @@ function createReader(onMessage, onPing, onClose) { // Per-frame length is capped above, but a stream of small continuation // frames that never sets fin would grow \`fragments\` without bound (60 MB // reassembled to 148 MB, verified). Cap the running sum too. - if (fragmentBytes > MAX_MESSAGE_BYTES) { + if (fragmentBytes > maxBytes) { onClose() return } @@ -248,7 +339,13 @@ function closePeer(peer, reason) { // up on it after two seconds), so the scrub cannot wait for it. A handoff // that is still running restores this by itself: every agent reconnect // re-sends its state, and in approval mode its screenshot. - if (peer.role === "agent") forgetPage() + if (peer.role === "agent") { + forgetPage() + // An agent that is gone will never drain, so its backpressure must not be + // what keeps the human muted: the handback they are about to send has to + // be read, held, and given to whichever agent connects next. + resumeHuman() + } // Detach the reader so a replaced client that ignores the close frame can no // longer feed route(); a lingering listener is how a peer keeps injecting. if (peer.read) peer.socket.removeListener("data", peer.read) @@ -288,7 +385,7 @@ function messageType(payload) { /** Keep what a human who joins late has to be shown, and drop what they must not. */ function rememberFromAgent(type, payload) { - if (type === "ended") { + if (type === MSG.ENDED) { // Terminal: keep the ending for a late human, drop everything that could // show the logged-in page to whoever opens the link next. lastEnded = payload @@ -301,9 +398,9 @@ function rememberFromAgent(type, payload) { // Forwarding that is harmless; storing it would put the page back in front // of the next visitor after this relay decided to drop it. if (humanEnded) return - if (type === "frame") lastFrame = payload - else if (type === "state") lastState = payload - else if (type === "focus") lastFocus = payload + if (type === MSG.FRAME) lastFrame = payload + else if (type === MSG.STATE) lastState = payload + else if (type === MSG.FOCUS) lastFocus = payload } /** One line per relay at most: a hostile client must not be able to fill the log. */ @@ -314,7 +411,7 @@ function logDrop(type) { dropLogged = true log("human message dropped", { type: String(type).slice(0, 32), - mode: MODE, + mode: HANDOFF_MODE, ended: humanEnded, }) } @@ -330,6 +427,19 @@ function acceptFromHuman(type, payload) { logDrop(type) return false } + // The scan floor, enforced here as well as in the core. The core's copy is + // what protects the browser from a screenshot loop; this one keeps a burst of + // accepted scans from costing a forward, a wake-up and a JSON parse on the + // agent's side for each one. Dropped, never queued: a scan is only worth + // anything against the page as it is now. + if (type === MSG.SCANQR) { + const now = Date.now() + if (now - lastScanAt < SCAN_INTERVAL_MS) { + logDrop(type) + return false + } + lastScanAt = now + } if (TERMINAL_HUMAN.has(type)) { // The human is done, for good. Buffer this for an agent that is // mid-reconnect, and stop replaying the last (logged-in) frame. @@ -353,7 +463,7 @@ function route(peer, payload, opcode) { return } const type = messageType(payload) - if (type === "ping") { + if (type === MSG.PING) { sendText(peer, PONG) return } @@ -361,8 +471,44 @@ function route(peer, payload, opcode) { else if (!acceptFromHuman(type, payload)) return // Newest frame wins: drop a frame bound for a backpressured receiver rather // than queue it in memory. Control and terminal messages are never dropped. - if (type === "frame" && other?.backpressure) return + if (type === MSG.FRAME && other?.backpressure) return write(other, payload, opcode) + // The human is producing faster than the agent's socket can take it. Stop + // reading that socket rather than growing this process's write queue with + // input nobody has asked for yet: TCP holds it, and the agent's \`drain\` + // starts it again. + // + // What this does and does not promise. The message just written is not held + // back, so the one that triggered the pause is delivered. Anything already + // behind it in the human's socket buffer — a handback among them — is *not + // lost but is delayed*, until the agent drains or goes away; \`closePeer\` + // resumes the human for the second case, so a dead agent cannot mute one + // forever. Separating terminal messages out would mean parsing before the + // flow-control decision, which is the work the 4 KiB cap exists to avoid. + if (peer.role === "human" && other?.backpressure) holdHuman(peer) +} + +/** + * Stop reading a human socket until the agent's has drained. + * + * Not a drop. Everything already read has been routed, and everything still in + * flight is where TCP is best at holding it. \`resumeHuman\` runs on the agent's + * \`drain\` and again when the agent goes away entirely, so a human is never + * left muted by a peer that is not coming back. + */ +function holdHuman(peer) { + if (peer.paused) return + peer.paused = true + peer.socket.pause() + log("human paused", { reason: "agent backpressure" }) +} + +function resumeHuman() { + const human = peers.get("human") + if (!human?.paused) return + human.paused = false + human.socket.resume() + log("human resumed", {}) } function log(event, detail) { @@ -377,6 +523,26 @@ function log(event, detail) { ) } +/** + * The mobile page, with this relay's two facts substituted in: which mode it + * is serving, and the wire vocabulary. The page never spells a message type + * itself — it reads \`MSG\` out of the same object the router above uses, so a + * type that is renamed in one place cannot survive in the other. + */ +function renderPage() { + // Function replacements, so a \`$&\` or a \`$'\` in a substituted value stays a + // literal instead of becoming a back-reference that rewrites the page. + const vocabulary = JSON.stringify({ + msg: MSG, + mode: MODE, + schemes: OPENABLE_SCHEMES, + }) + return PAGE.replace("__HANDRAISE_MODE__", () => HANDOFF_MODE).replace( + "__HANDRAISE_VOCAB__", + () => vocabulary, + ) +} + const server = createServer((req, res) => { const url = new URL(req.url || "/", "http://relay") if (url.pathname === "/healthz") { @@ -392,9 +558,9 @@ const server = createServer((req, res) => { "content-type": "text/html; charset=utf-8", "cache-control": "no-store", }) - // MODE is one of two literals, so this substitution can only produce the - // two pages this file was written for. - res.end(PAGE.replace("__HANDRAISE_MODE__", MODE)) + // HANDOFF_MODE is one of two literals, so this substitution can only + // produce the two pages this file was written for. + res.end(renderPage()) return } res.writeHead(404, { @@ -454,7 +620,14 @@ server.on("upgrade", (req, socket, head) => { const previous = peers.get(role) if (previous) closePeer(previous, "replaced") - const peer = { role, socket, open: true, backpressure: false } + const peer = { + role, + socket, + open: true, + backpressure: false, + // Human only: set while its socket is held for a backpressured agent. + paused: false, + } peers.set(role, peer) log("peer connected", { role }) @@ -462,12 +635,15 @@ server.on("upgrade", (req, socket, head) => { (payload, opcode) => route(peer, payload, opcode), (payload) => write(peer, payload, OP_PONG), () => closePeer(peer, "peer closed the socket"), + role === "human" ? MAX_HUMAN_MESSAGE_BYTES : MAX_MESSAGE_BYTES, ) peer.read = read if (head?.length) read(head) socket.on("data", read) socket.on("drain", () => { peer.backpressure = false + // The agent can take input again, so the human may speak again. + if (peer.role === "agent") resumeHuman() }) socket.on("error", () => closePeer(peer, "socket error")) socket.on("close", () => closePeer(peer, "socket closed")) @@ -503,7 +679,7 @@ server.listen(PORT, "0.0.0.0", () => { // Report the bound port, not the requested one: port 0 asks the OS to pick a // free one, which is how the local test suite avoids fighting for 3000. const bound = server.address() - log("relay listening", { port: bound?.port ?? PORT, mode: MODE }) + log("relay listening", { port: bound?.port ?? PORT, mode: HANDOFF_MODE }) }) const PAGE = \` @@ -783,14 +959,18 @@ const PAGE = \` or one step and sit together in typing order; clear destroys the whole field with no undo, so it is a word rather than a glyph a stranger has to guess at, and it sits behind a gutter the thumb has to reach for. A missed - backspace can no longer empty the field. */ - #key-clear { + backspace can no longer empty the field. + + Scan QR is past the gutter with it — not because it is destructive, but + because it is not a key. It asks the agent a question about the page + instead of typing into it, and the three glyphs keep their own group. */ + #key-clear, #key-qr { flex: 0 0 auto; min-width: 44px; - margin-left: 18px; padding: 0 8px; font-size: 13px; } + #key-clear { margin-left: 18px; } .key:active:not(:disabled) { color: var(--text); border-color: oklch(0.44 0 0); @@ -898,6 +1078,90 @@ const PAGE = \` #overlay[hidden] { display: none; } #overlay h1 { margin: 0; font-size: 20px; letter-spacing: -0.02em; } #overlay p { margin: 0; color: var(--muted); font-size: 14px; } + /* What the QR code said. + A sheet and not a second overlay: the overlay ends the session, this one is + an answer the human reads and dismisses, and the frame stays behind it + because the next thing they do is usually on the page. It rises from the + bottom edge, where the thumb already is. */ + #sheet { + position: fixed; + inset: 0; + /* Above the ending overlay, which is 10. The overlay is opaque, and a + handback taken before the human tapped Open would otherwise bury the + link this whole feature exists to deliver — with the button disabled, + the socket closing and no way to scan again. The sheet has its own Done + button, so the end screen is one tap away. */ + z-index: 11; + display: flex; + align-items: flex-end; + background: oklch(0.11 0 0 / 0.72); + } + #sheet[hidden] { display: none; } + #sheet-card { + width: 100%; + max-height: 80%; + overflow-y: auto; + display: flex; + flex-direction: column; + gap: 12px; + padding: 18px calc(16px + env(safe-area-inset-right)) calc(16px + env(safe-area-inset-bottom)) calc(16px + env(safe-area-inset-left)); + border-top: 1px solid var(--line); + border-radius: var(--radius) var(--radius) 0 0; + background: var(--surface); + transform: translateY(0); + transition: transform 220ms cubic-bezier(0.23, 1, 0.32, 1); + } + @starting-style { + #sheet-card { transform: translateY(100%); } + } + #sheet-title { margin: 0; font-size: 17px; letter-spacing: -0.02em; } + #sheet-links { display: flex; flex-direction: column; gap: 12px; } + .link { + display: flex; + flex-direction: column; + gap: 8px; + padding: 12px; + border: 1px solid var(--line); + border-radius: var(--radius); + background: var(--bg); + } + /* The link is the thing being decided on, so it is shown in full and it + wraps. anywhere, because a token has no spaces to break at — a truncated + URL is exactly how somebody is talked into opening the wrong one. */ + .link-text { + margin: 0; + font-size: 13px; + line-height: 1.35; + color: var(--text); + overflow-wrap: anywhere; + } + /* The host is the one word that answers "whose site is this". The rest of a + URL is a token nobody reads, so it stays muted and the host does not. */ + .link-host { color: var(--text); font-weight: 600; } + .link-text > span:not(.link-host) { color: var(--muted); } + .link-note { margin: 0; font-size: 12px; color: var(--muted); } + .link-actions { display: flex; gap: 8px; } + /* Same box for the anchor and the button, so the row does not shift by a + pixel between a link that can be opened and one that can only be copied. */ + .link-action { + flex: 1 1 0; + min-height: 44px; + display: flex; + align-items: center; + justify-content: center; + padding: 0 12px; + border: 1px solid var(--field); + border-radius: var(--radius); + background: transparent; + color: var(--text); + font: inherit; + font-size: 15px; + font-weight: 500; + text-decoration: none; + } + .link-action:active { background: oklch(0.26 0 0 / 0.4); } + #sheet-close { min-height: 44px; } + .empty { margin: 0; color: var(--muted); font-size: 14px; } /* One page, two jobs. The relay bakes the mode into the body, and the controls belonging to the other job are gone rather than disabled: the relay refuses to route what they would have sent anyway. */ @@ -934,6 +1198,10 @@ const PAGE = \` .ghost::before { display: none; } .ghost[data-holding] { background: oklch(0.65 0.2 25 / 0.14); } #approve[data-holding] { background: oklch(0.985 0 0 / 0.12); } + #sheet-card { transition: none; } + @starting-style { + #sheet-card { transform: none; } + } #overlay { transition: opacity 150ms linear; } @starting-style { #overlay { opacity: 0; transform: none; } @@ -967,6 +1235,7 @@ const PAGE = \` +

@@ -987,6 +1256,13 @@ const PAGE = \`

+