Screen Reader Speech Generation Strategy

[NeverMorewd]

Screen Reader Speech Generation Pipeline

Overview

The goal of this subsystem is to transform a raw UI Automation (UIA) element into a meaningful spoken description that helps users understand:

• What the element is

• What the element does

• Where the element belongs

• Whether the element is actionable

• Additional contextual information

• Fallback information when accessibility data is incomplete

The system should not simply read the UIA Name property.

Instead, it should build a semantic description from multiple accessibility and contextual sources.

---

Design Goals

Primary Goals

• Produce meaningful speech for most UI elements

• Recover gracefully when accessibility information is missing

• Support application-specific enhancements

• Support configurable verbosity levels

• Support future OCR integration

Non-Goals

• Full screen understanding

• AI-based UI reasoning

• Replacing UI Automation

---

Pipeline

UIA Element
    │
    ▼
Element Analyzer
    │
    ├─ Accessibility Data
    ├─ Context Analysis
    ├─ Rule Engine
    ├─ OCR Fallback
    │
    ▼
Speech Description
    │
    ▼
TTS Provider

---

Data Collection

Core UIA Properties

Collect all available information from the target element.

Name
ControlType
LocalizedControlType
AutomationId
HelpText
FrameworkId
BoundingRectangle
IsEnabled
IsOffscreen

Patterns

Attempt to retrieve:

ValuePattern
TogglePattern
SelectionItemPattern
ExpandCollapsePattern
RangeValuePattern
InvokePattern

Legacy Accessibility

Fallback to:

IAccessible.accName
IAccessible.accDescription
IAccessible.accValue

---

Speech Model

public sealed class SpeechDescription
{
    public string? Label { get; init; }
public string? ControlType { get; init; }

public string? State { get; init; }

public string? Context { get; init; }

public string? Tooltip { get; init; }

public string? Warning { get; init; }

public double Confidence { get; init; }

}

Example:

{

"label": "Google Chrome",

"controlType": "button",

"state": "running",

"context": "Windows taskbar",

"tooltip": "Switch to Google Chrome",

"confidence": 0.98

}

---

Rule Engine

The speech generation system should be based on independent rules.

Each rule contributes information to the final speech description.

TaskbarRule

ButtonRule

CheckBoxRule

MenuRule

TooltipRule

AdminRule

OCRRule

Rules should be composable and independent.

---

Control Type Rules

Button

Input:

Name = Open

ControlType = Button

Output:

Open button

---

CheckBox

Input:

Name = Remember me

Checked = true

Output:

Remember me checkbox, checked

---

RadioButton

Output:

Dark theme radio button, selected

---

Edit Box

Read:

• Name

• Current Value

• ReadOnly State

Output:

Search edit box, text Visual Studio

---

Slider

Output:

Volume slider, 70 percent

---

Progress Bar

Output:

Download progress, 45 percent

---

Tree View Item

Output:

Documents tree item, expanded

---

Menu Item

Output:

Save menu item

or

Save menu item in File menu

---

Tab Item

Output:

Settings tab, selected

---

Context Analysis

Users often need context to understand an element.

Example:

Without context:

Open button

With context:

Open button in File Explorer toolbar

Parent Hierarchy Analysis

Walk up the UIA tree.

Example:

Button

└ Toolbar

└ File Explorer

Generated context:

File Explorer toolbar

---

Taskbar Recognition

Detect taskbar elements.

Possible indicators:

Shell_TrayWnd

MSTaskListWClass

Explorer.exe

Example output:

Google Chrome button on Windows taskbar

Additional state:

Running

Minimized

Active

Example:

Visual Studio button on Windows taskbar, active

---

Tooltip Integration

Tooltips often contain valuable descriptions.

Sources:

UIA HelpText

Tooltip Window

IAccessible Description

Example:

Refresh button. Reload current page.

Avoid duplicated speech.

Bad:

Refresh button. Refresh.

Good:

Refresh button. Reload current page.

---

Permission Awareness

Some applications expose limited accessibility information when running elevated.

Detect:

Target Process Elevated

Current Process Not Elevated

Speech:

Limited accessibility information available.

Administrator privileges may be required.

Example:

Visual Studio window.

Administrator privileges may be required.

---

Empty Name Recovery

Many applications expose controls without accessible names.

A fallback chain should be used.

Priority Order

1. Name

UIA Name

2. Value Pattern

ValuePattern.Value

3. Legacy Accessibility

accName

accDescription

accValue

4. Associated Label Detection

Example:

Username

[Edit Box]

Output:

Username edit box

5. Parent Context

Example:

Unnamed button in Visual Studio

6. Process Name

Example:

Unnamed element in explorer.exe

7. Window Title

Example:

Unnamed element in Settings

8. Position Description

Example:

Unnamed element near top left corner

9. OCR

Capture:

BoundingRectangle

Perform OCR.

Output:

Download

Speech:

Download, detected from screen text

---

OCR Integration

OCR should be considered a last resort.

Conditions:

Name Missing

AND

No Value Pattern

AND

No Label Association

Suggested workflow:

Capture Element Bounds

│

▼

OCR Engine

│

▼

Text Candidate

│

▼

Speech Builder

OCR-derived content should always be identified as inferred content.

---

Confidence Scoring

Each generated speech description should include a confidence score.

Source	Confidence
UIA Name	100%
Value Pattern	95%
HelpText	90%
Label Association	85%
Parent Context	70%
OCR	50%
Position Only	20%

This score can determine:

• Whether OCR is necessary

• Whether speech should be cached

• Whether diagnostic logging is required

---

Verbosity Levels

Compact

Minimal information.

Example:

Open button

---

Normal

Default mode.

Example:

Open button in File Explorer toolbar

---

Detailed

Include all available information.

Example:

Open button.
File Explorer toolbar.
Shortcut Control O.
Enabled.

---

Application-Specific Extensions

Support application-specific speech enhancements.

Examples:

Windows Taskbar
Visual Studio
Chrome
Edge
Office
Explorer

Architecture:

public interface IElementSpeechRule
{
    bool CanHandle(ElementContext context);
void Apply(
    ElementContext context,
    SpeechDescriptionBuilder builder);

}

Examples:

TaskbarSpeechRule

ChromeSpeechRule

ExplorerSpeechRule

VisualStudioSpeechRule

These rules can be loaded dynamically and registered through dependency injection.

---

Example Outputs

Example 1

Input:

Button

Name = Close

Output:

Close button

---

Example 2

Input:

Taskbar Button

Name = Visual Studio

Output:

Visual Studio button.

Windows taskbar.

Running.

---

Example 3

Input:

Edit Box

Name = Search

Value = Accessibility

Output:

Search edit box.

Text Accessibility.

---

Example 4

Input:

Name Missing

OCR Result = Download

Output:

Download.

Detected from screen text.

---

Future Enhancements

• AI-assisted element summarization

• Browser DOM integration

• Live region support

• Table navigation support

• Accessibility diagnostics mode

• User-customizable speech templates

• Multilingual speech generation

• Screen understanding based on OCR + UIA fusion