DesktopKit

Playwright-like testing tool for Linux desktop applications.

DesktopKit lets you automate and test GTK/GNOME desktop apps with a familiar, modern API — the same way Playwright works for browsers. It includes an MCP server so AI assistants can drive desktop apps directly.

from desktopkit import Application, expect

with Application.launch("gnome-calculator") as app:
    app.get_by_role("button", name="2").click()
    app.get_by_role("button", name="+").click()
    app.get_by_role("button", name="3").click()
    app.get_by_role("button", name="=").click()

    expect(app.get_by_role("label", name="5")).to_be_visible()

Features

• Playwright-style API — get_by_role(), get_by_name(), get_by_label() locators
• Auto-waiting — every action waits for the element to be visible and enabled
• Smart assertions — expect(element).to_be_visible() with auto-retry
• MCP server — AI agents can launch apps, click elements, and inspect UI trees
• CLI inspector — desktopkit inspect Calculator dumps the accessibility tree
• Headless mode — desktopkit run --headless wraps tests in Xvfb for CI
• Screenshot capture — capture window or element screenshots

Requirements

• Linux with AT-SPI accessibility framework enabled
• Python 3.10+
• PyGObject (gi.repository.Atspi)
• Applications must have accessibility support (GTK, Qt with AT-SPI bridge)

Installation

pip install -e .

# Or with dev dependencies
pip install -e ".[dev]"

System dependencies (Debian/Ubuntu)

sudo apt install \
    python3-gi \
    gir1.2-atspi-2.0 \
    libatk1.0-0 \
    at-spi2-core \
    libgirepository-2.0-dev \
    xvfb              # for headless mode

Quick Start

1. Write a test

from desktopkit import Application, expect

def test_calculator():
    with Application.launch("gnome-calculator") as app:
        app.get_by_role("button", name="1").click()
        app.get_by_role("button", name="+").click()
        app.get_by_role("button", name="1").click()
        app.get_by_role("button", name="=").click()

        expect(app.get_by_role("label", name="2")).to_be_visible()

2. Run the test

# With display
pytest tests/ -v

# Headless (CI)
desktopkit run --headless tests/

3. Inspect an app's UI tree

# Attach to a running app
desktopkit inspect Calculator

# Launch and inspect
desktopkit inspect --launch gnome-calculator

# JSON output
desktopkit inspect Calculator --json --depth 3

API Reference

Application

# Launch a new app
app = Application.launch("gnome-calculator")
app = Application.launch("gnome-calculator", timeout=15)

# Attach to a running app
app = Application.attach("Calculator")

# Context manager (auto-close)
with Application.launch("gnome-calculator") as app:
    ...

# Inspect the UI tree
print(app.format_tree(max_depth=3))
tree_dict = app.dump_tree()

# Close
app.close()

Locators

Locators are lazy — they re-query the AT-SPI tree on every action.

# By role (+ optional name)
button = app.get_by_role("button", name="OK")

# By accessible name
element = app.get_by_name("result-display")

# By label text
field = app.get_by_label("Username")

# Chaining
dialog = app.get_by_role("dialog", name="Preferences")
save_btn = dialog.get_by_role("button", name="Save")

# Multiple elements
all_buttons = app.get_by_role("button").all()
first_button = app.get_by_role("button").first()
third_button = app.get_by_role("button").nth(2)
count = app.get_by_role("button").count()

Actions

All actions auto-wait for the element to be visible and enabled.

locator.click()
locator.double_click()
locator.type_text("hello world")
locator.clear()
locator.press_key("Return")
locator.focus()

Assertions

Assertions auto-retry until the condition is met or timeout expires.

from desktopkit import expect

expect(locator).to_be_visible()
expect(locator).to_be_enabled()
expect(locator).to_be_hidden()
expect(locator).to_be_focused()
expect(locator).to_be_checked()
expect(locator).to_have_text("Hello")
expect(locator).to_have_text(re.compile(r"\d+"))
expect(locator).to_have_name("OK")
expect(locator).to_have_count(3)

# Negation
expect(locator).not_to.to_be_visible()
expect(locator).not_to.to_have_text("Error")

# Custom timeout
expect(locator, timeout=10).to_be_visible()

Available Roles

Common roles for get_by_role():

Role	Description
button	Push button
label	Text label
text	Text display area
entry	Text input field
check_box	Checkbox
radio_button	Radio button
combo_box	Dropdown/combobox
menu	Menu container
menu_item	Menu item
menu_bar	Menu bar
dialog	Dialog window
frame	Application window frame
panel	Container panel
tab	Tab in a tab list
list	List container
list_item	Item in a list
tree	Tree view
tree_item	Item in a tree
slider	Slider control
progress_bar	Progress indicator
scroll_bar	Scroll bar
tool_bar	Toolbar
status_bar	Status bar

Aliases also work: textbox, input, dropdown, checkbox, radio, tab.

MCP Server

DesktopKit includes an MCP (Model Context Protocol) server that lets AI assistants interact with desktop applications.

Setup

Add to your MCP client configuration (Claude Desktop, VS Code, etc.):

{
	"mcpServers": {
		"desktopkit": {
			"command": "desktopkit-mcp",
			"env": {
				"DISPLAY": ":0"
			}
		}
	}
}

Available MCP Tools

Tool	Description
launch_app	Launch a desktop application
attach_app	Attach to a running application
close_app	Close a managed application
list_apps	List all managed applications
get_ui_tree	Get the accessibility tree as JSON
find_element	Find an element by role/name
click_element	Click an element
type_text	Type text into an element
press_key	Simulate a key press
take_screenshot	Capture a window screenshot
assert_element	Assert element state (visible, enabled, has_text, etc.)

MCP Resources

Resource	Description
desktopkit://ui-tree/{app}	Current UI tree as JSON

MCP Prompts

Prompt	Description
write-desktop-test	Generate a test script for an app
debug-element	Help find a hard-to-locate element

Example AI Interaction

Once configured, you can tell your AI assistant:

"Launch gnome-calculator, click 7, then +, then 3, then =, and tell me what the result display shows."

The AI will use the MCP tools to:

1. launch_app("gnome-calculator")
2. get_ui_tree("gnome-calculator") to discover element names
3. click_element(app_name="gnome-calculator", role="button", name="7")
4. ...and so on

CLI

# Inspect an app's accessibility tree
desktopkit inspect Calculator
desktopkit inspect --launch gnome-calculator
desktopkit inspect Calculator --json --depth 3

# Run tests
desktopkit run tests/
desktopkit run --headless tests/
desktopkit run --headless --display :42 tests/test_app.py

Project Structure

src/desktopkit/
├── __init__.py          # Public API: Application, Element, Locator, expect
├── _app.py              # Application class (launch, attach, close)
├── _element.py          # Element class (click, type_text, get_text, etc.)
├── _locator.py          # Locator class (lazy resolution, chaining)
├── _waiter.py           # Auto-waiting engine (poll until actionable)
├── _assertions.py       # expect() with auto-retry assertions
├── _screenshot.py       # Screenshot capture
├── _atspi/
│   ├── backend.py       # AT-SPI initialization and low-level operations
│   ├── tree.py          # UI tree traversal, search, dump
│   └── roles.py         # Role name mappings
├── _mcp/
│   └── server.py        # MCP server with tools, resources, prompts
└── _cli/
    ├── main.py          # CLI entry point
    ├── inspect_cmd.py   # `desktopkit inspect` command
    └── run_cmd.py       # `desktopkit run` command

License

MIT