Desktop Control V2

Desktop Control V2 turns a background Windows application into structured scene JSON. It combines application object models, accessibility, native controls, and graphics draw calls. The agent never receives a screenshot and cannot request an absolute-coordinate click.

The user still gets a live preview window. That preview rasterizes the isolated desktop in a separate process and overlays the currently selected scene node plus a teal cursor. Preview pixels never enter MCP results.

Contract

application / UIA / Win32 / graphics adapters
                    |
                    v
       desktop-control.scene/v1 JSON
       nodes + logical geometry + state
       stable IDs + declared capabilities
                    |
                    v
      find(node selector) -> act(node ID)

human-only HWND compositor ----------------> preview window

The public tools are windows_scene_attach, windows_scene_refresh, windows_scene_snapshot, windows_scene_find, and windows_scene_action. There is no observation-image tool and no generic click tool.

Adapters

Tier	Examples	What it contributes
Application	Multisim DrwEngine, .ms14; Minecraft menu/narrator bridge	Names, logical objects, topology, commands
Accessibility	UI Automation, MSAA	Roles, names, values, patterns
Native	HWND controls, command IDs	Dialog/list/menu state and allowlisted actions
Render	GDI, Direct2D/DirectWrite, OpenGL, D3D11/12, Vulkan	Draw groups, text, paths, meshes, transforms, materials

The native collector builds for x86 and x64. Its live x86 Multisim probe captures GDI draw imports and includes version-guarded DrwEngine object/store detours for deeper traversal. The x64 Bedrock probe captures D3D12 command recording/submission, decodes half-float UI vertices, and preserves retained menu geometry when the game reuses a composite texture. Both emit JSONL events and never copy a framebuffer.

Preview and interactive commands keep named-pipe connections open for the lifetime of the client. This avoids reconnect logging and lets node actions drive a hidden desktop with semantic keys while the preview independently renders for the human.

Install

Requirements: Windows 10/11, Node.js, pnpm, Python 3.12, Visual Studio C++ x86/x64 build tools, and the Microsoft.GameInput NuGet headers.

Existing v0.x releases are preserved V1 binaries. Desktop Control V2 currently installs from source.

git clone https://github.com/aShanki/desktop-control-system.git
cd desktop-control-system
pnpm install --frozen-lockfile
py -3.12 -m pip install -r runtime/requirements.txt
pnpm build:native
pnpm check
pnpm test
pnpm build

The V1 hidden-desktop host and agent are bundled under runtime/; a sibling checkout is no longer required. Connect the stdio MCP server with pnpm --dir C:\path\to\desktop-control-system start.

Install the portable Agent Skill for Codex, Claude Code, Cursor, GitHub Copilot, or the cross-agent .agents/skills standard:

.\scripts\install-agent-skill.ps1 -Agent universal -Scope user

See installation, usage, architecture, Multisim acceptance test, and Minecraft Bedrock menu adapter.