Skip to content

feat: uiSnapshot() — the accessibility tree plus the text it misses - #14

Merged
woladi merged 1 commit into
mainfrom
feat/ui-snapshot
Aug 24, 2026
Merged

woladi merged 1 commit into
mainfrom
feat/ui-snapshot

Conversation

@woladi

@woladi woladi commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Completes docs/BOX-MODEL.md phase 4. Captures a window once, walks its accessibility tree, OCRs the same region, and reports the visible text no node accounts for.

const snap = await uiSnapshot({ app: 'MyApp' });
snap.summary;      // { nodes: 491, labelled: 402, ocrBlocks: 123, unresolved: 21, axTextCoverage: 0.83 }
snap.unresolved[0] // { text: "Sprzedaż Q4", box: [420,300,88,16], confidence: 0.98, coveredByNode: 17 }

unresolved does double duty: it completes the box model where AX is blind (canvas, WebGL, games, images with text baked in), and every entry is an accessibility gap in the app under test — coveredByNode present means an unlabelled control, absent means nothing exposed at all.

Two things the first working version got wrong

The coverage metric lied when the walk was capped. axTextCoverage read 0.34 on a Safari window at maxElements: 200 and 0.83 for the same window walked completely. The first figure measures our own budget, not the app's accessibility — but it reads like a verdict on the app. It is now null whenever budget.capped is true, with cappedWalk: true alongside.

The capture and the tree each resolved their own target. Given a pid and no app name, colours and OCR came from the whole display while the geometry came from one window. The window is now resolved once and both use it.

Verification

104/104 tests. The merge is a pure function over (tree, OCR blocks, frame), so its 13 tests need no screen, no permissions and no helper — they run in milliseconds anywhere, including CI. Includes a guard that the whitespace class does not swallow ASCII, since this file needed its own copy of the normalizer and that bug has bitten this codebase before.

🤖 Generated with Claude Code

Captures a window once, walks its accessibility tree, runs OCR over the
same region, and reports every piece of visible text no node accounts
for. That completes the box model for anything custom-drawn — canvas,
WebGL, games, images with text baked in — where AX is simply blind, and
each entry is also an accessibility gap in the app under test:
coveredByNode present means a control is there but unlabelled, absent
means nothing is exposed at all.

Two things the first working version got wrong:

The coverage metric lied when the walk was capped. axTextCoverage read
0.34 on a Safari window at maxElements 200 and 0.83 for the same window
walked completely — the first figure measures our own budget, not the
app's accessibility, but it reads like a verdict on the app. It is now
null whenever budget.capped is true, with cappedWalk alongside it.

The capture and the tree each resolved their own target, so with a pid
and no app name the colours and OCR came from the whole display while the
geometry came from one window. The window is now resolved once and both
use it.

The merge is a pure function over a tree, OCR blocks and a frame, so its
13 tests need no screen, no permissions and no helper — they run in
milliseconds anywhere. Includes a guard that the whitespace class does not
swallow ASCII, since this file needed its own copy of the normalizer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@woladi
woladi merged commit 50ee48d into main Aug 24, 2026
1 check passed
@woladi
woladi deleted the feat/ui-snapshot branch August 24, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant