feat: uiSnapshot() — the accessibility tree plus the text it misses - #14
Merged
Merged
Conversation
Captures a window once, walks its accessibility tree, runs OCR over the same region, and reports every piece of visible text no node accounts for. That completes the box model for anything custom-drawn — canvas, WebGL, games, images with text baked in — where AX is simply blind, and each entry is also an accessibility gap in the app under test: coveredByNode present means a control is there but unlabelled, absent means nothing is exposed at all. Two things the first working version got wrong: The coverage metric lied when the walk was capped. axTextCoverage read 0.34 on a Safari window at maxElements 200 and 0.83 for the same window walked completely — the first figure measures our own budget, not the app's accessibility, but it reads like a verdict on the app. It is now null whenever budget.capped is true, with cappedWalk alongside it. The capture and the tree each resolved their own target, so with a pid and no app name the colours and OCR came from the whole display while the geometry came from one window. The window is now resolved once and both use it. The merge is a pure function over a tree, OCR blocks and a frame, so its 13 tests need no screen, no permissions and no helper — they run in milliseconds anywhere. Includes a guard that the whitespace class does not swallow ASCII, since this file needed its own copy of the normalizer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes
docs/BOX-MODEL.mdphase 4. Captures a window once, walks its accessibility tree, OCRs the same region, and reports the visible text no node accounts for.unresolveddoes double duty: it completes the box model where AX is blind (canvas, WebGL, games, images with text baked in), and every entry is an accessibility gap in the app under test —coveredByNodepresent means an unlabelled control, absent means nothing exposed at all.Two things the first working version got wrong
The coverage metric lied when the walk was capped.
axTextCoverageread 0.34 on a Safari window atmaxElements: 200and 0.83 for the same window walked completely. The first figure measures our own budget, not the app's accessibility — but it reads like a verdict on the app. It is nownullwheneverbudget.cappedis true, withcappedWalk: truealongside.The capture and the tree each resolved their own target. Given a
pidand no app name, colours and OCR came from the whole display while the geometry came from one window. The window is now resolved once and both use it.Verification
104/104 tests. The merge is a pure function over (tree, OCR blocks, frame), so its 13 tests need no screen, no permissions and no helper — they run in milliseconds anywhere, including CI. Includes a guard that the whitespace class does not swallow ASCII, since this file needed its own copy of the normalizer and that bug has bitten this codebase before.
🤖 Generated with Claude Code