Skip to content
This repository was archived by the owner on Nov 12, 2026. It is now read-only.

feat(mac): the document as a text area VoiceOver and agents read and edit as text - #24

Merged
adammharris merged 3 commits into
mainfrom
text-accessibility
Oct 23, 2026
Merged

adammharris merged 3 commits into
mainfrom
text-accessibility

Conversation

@adammharris

@adammharris adammharris commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

The Mac app's document reached the accessibility tree only as AXStaticText: no value, selection, caret or insertion through AX, so an agent could type only with raw key events and could not check them. Now it is an AXTextArea.

Uses diaryx-org/blitz#15 (merged into leaf): a hook in blitz-shell through which the embedder changes each tree built. apps/leaf-dioxus takes Blitz from branch = "leaf", locked at 9ee8db17.

What it does

  • Editor::describe (crates/leaf-dioxus/src/access.rs) makes the editor's element (id="leaf-text") a MultilineTextInput: its value is the whole text, and its children are text runs, one per visual line (at most 255 characters, chained on the line), with character lengths (grapheme clusters), positions, widths, word starts, and the selection.
  • The text is the text as shown in the rendered view, a line per row (rows that hold no text, such as table rules and block gaps, are left out). In the source view it is the Markdown. Positions map to the source through the view's own (row, column) map; a range ends just past its last character, so selecting a bold word does not take its **.
  • Editor::accessibility_action carries out SetTextSelection, ReplaceSelectedText and SetValue through the document as typing does (select_range + insert, the path that ends in splice_exact). A new value replaces only the stretch that differs; in the source view it is written exactly (replace_source).
  • The app (apps/leaf-dioxus/src/access.rs) installs the hook on every window, replaces the nodes under the editor with the runs, gives the editor the focus unless a page control has it, and routes requests for the text area or its runs to the editor.
  • The text and runs are cached while the document, caret and layouts hold: about 1 ms a frame on a 120,000-character document (debug build); a rebuild after an edit, about 14 ms there.

Checked on the app (background window, Cua Driver and the AX API)

  • AXRole AXTextArea; AXValue, AXNumberOfCharacters, AXSelectedText, AXSelectedTextRange, AXInsertionPointLineNumber, AXRangeForLine, AXLineForIndex, AXStringForRange and AXBoundsForRange all answer. Each soft-wrapped visual line is its own line. Per-character bounds are exact (bold → 33.7 pt wide).
  • Setting AXSelectedTextRange selects, and the drawn selection follows. Cua Driver's set_value on the text area's element token edited the document, and reading the value back showed the edit. Toolbar presses still work.
  • Source view: the value is the Markdown, and selection and lines map.
  • Not yet checked with VoiceOver itself. That needs VoiceOver running with its AppleScript control turned on, and it takes over speech and keys.

Tests

12 unit tests in crates/leaf-dioxus/src/access.rs: shown and source text, grapheme clusters, selection mapping, caret at a line break, soft wraps and long lines, select, replace and set-value edits, and an empty document. ship ci dioxus and ship ci versions pass.

… Editor::describe

The editor draws its document as rows of runs, which Blitz's tree gives as
static text, with no value, selection or caret. `Editor::describe` makes the
editor's node (the element `TEXT_ID` names) an AccessKit
`MultilineTextInput`: its value is the whole text, and its children are text
runs, one per visual line (and at most 255 characters, chained on the line),
with each character's length, position and width, where its words start, and
the selection. The text is the text as shown in the rendered view, its rows
of drawn glyphs, a line per row; in the source view it is the Markdown. A
character is a grapheme cluster. Positions map to the source through the
view's own (row, column) map, a range's end just past its last character,
so selecting a bold word does not take its closing `**`.

`Editor::accessibility_action` carries out what assistive tech asks of the
text area or a run, through the document as typing does: SetTextSelection
selects the range's source, ReplaceSelectedText types over the selection,
and SetValue types the stretch that differs over it (in the source view the
value is written exactly, by `replace_source`). The text and its runs are
kept while the document, the caret and the rows' layouts are unchanged.

Behavioural-change: the element the editor draws its rows in has the DOM id `leaf-text`.
…edit as text

Every accessibility tree built for a window passes through a hook
(blitz-shell's `View::set_accessibility_tree_hook`, from diaryx-org/blitz's
branch `a11y-tree-hook` until diaryx-org/blitz#15 merges), which hands the
editor's element to `Editor::describe` and puts its text runs in place of
the nodes under it. The editor has the focus in the tree unless one of the
page's controls has it. A request for the text area or one of its runs goes
to `Editor::accessibility_action`; a press still becomes a click.

Checked on the app with Cua Driver and the AX API, in a background window:
AXValue, AXNumberOfCharacters, AXSelectedText(Range),
AXInsertionPointLineNumber, AXRangeForLine, AXLineForIndex,
AXStringForRange and AXBoundsForRange answer, a line a visual line; setting
AXSelectedTextRange moves the drawn selection, and setting AXValue edits the
document.

Behavioural-change: the document is an AXTextArea whose value is its text, and setting its value or selected range edits and selects in it.
@adammharris
adammharris merged commit e0d29f4 into main Oct 23, 2026
6 checks passed
@adammharris
adammharris deleted the text-accessibility branch October 30, 2026 01:24
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant