Skip to content

feat(windows): add native Computer Use helper - #5

Draft
sunheyi6 wants to merge 2 commits into
maka-agent:maka/basefrom
sunheyi6:codex/windows-dotnet-helper
Draft

feat(windows): add native Computer Use helper#5
sunheyi6 wants to merge 2 commits into
maka-agent:maka/basefrom
sunheyi6:codex/windows-dotnet-helper

Conversation

@sunheyi6

@sunheyi6 sunheyi6 commented Sep 1, 2026

Copy link
Copy Markdown

Why

Apache Maka issue #4318 needs a native Windows Computer Use executor with explicit window authority, bounded lifecycle behavior, and evidence that does not depend on global input or screen-rectangle capture.

What changes

  • add a supervised C#/.NET helper using the private maka.cu.windows/0 JSON-RPC protocol
  • enumerate top-level windows and require an explicit HWND for observation
  • expose bounded UI Automation snapshots with one-use opaque element tokens
  • implement verified set_value and click_element through UIA patterns
  • capture the selected window through Windows Graphics Capture CreateForWindow
  • revalidate PID, process start time, HWND, window generation, and UIA identity before mutation
  • add cancellation, backpressure, parent-death, blocked-provider, restart, window-recreation, and control-replacement coverage
  • add a self-contained win-x64 publish script and manifest; artifacts remain distributionReady: false pending release ownership

This PR intentionally keeps the existing Go CLI/MCP implementation unchanged. The new helper is a separate supervised product process.

Validation

  • dotnet build for helper and WinForms fixture: 0 warnings, 0 errors
  • go test ./...
  • self-contained win-x64 publish
  • protocol regression suite: 4/4
  • interactive lifecycle suite: all checks passed, including WGC occlusion, cancel settlement, helper restart, parent exit, whole-window recreation, and same-window control replacement
  • real host integration from the companion Maka branch: observe with screenshot, set_value, and re-observe verification

Related: apache/maka#4318
Companion host integration: apache/maka#4409.

中文说明

此 PR 在 maka-cu 中加入 Windows 原生 Computer Use helper:使用 UI Automation 观察和执行语义操作,使用 Windows Graphics Capture 对指定窗口截图,并通过一次性快照、窗口身份校验、取消、父进程退出和 helper 重启机制避免错误操作与静默重试。

现有 Go CLI/MCP 实现保持不变。发布产物目前仍标记为 distributionReady: false,等待上游确定正式构建、签名和分发责任。

关联 Apache Maka issue:apache/maka#4318

@sunheyi6

sunheyi6 commented Sep 1, 2026

Copy link
Copy Markdown
Author

@liugddx @M4n5ter — this is the proposed long-term native code home from apache/maka#4318, paired with apache/maka#4409.

Please review whether targeting maka/base, retaining the existing Go CLI/MCP surface, and adding the C# helper as a separately supervised process match the intended ownership boundary. The interactive lifecycle suite and the real Maka host integration both pass; release provenance and clean-machine qualification remain explicit gates.

中文说明

这是 apache/maka#4318 建议的原生代码长期归属,并与 apache/maka#4409 配套。

请确认目标分支 maka/base、保留现有 Go CLI/MCP,以及把 C# helper 作为独立监管进程加入是否符合预期。交互式生命周期和真实 Maka Host 联调均已通过;正式产物来源和干净机器验收仍是明确门禁。

@sunheyi6

sunheyi6 commented Sep 1, 2026

Copy link
Copy Markdown
Author

Local Desktop acceptance clarified the publish contract: PublishSingleFile contains the managed payload, but Windows Desktop/WPF still emits five required native companion DLLs. Copying only the exe can complete initialize and then fail when UI Automation initializes.

Commit 96351b8 updates the native manifest to hash and size the complete publish closure and documents the required companions. The paired Host PR now copies, packages, and verifies that exact directory.

中文说明

本地 Desktop 验收确认:PublishSingleFile 只表示托管 payload 单文件,Windows Desktop/WPF 仍会输出五个必需原生 DLL。只复制 exe 可能握手成功,但在 UIA 初始化时失败。

提交 96351b8 已让原生 manifest 记录整个发布闭包,并明确 companion DLL 要求;配套 Host PR 会整体复制、打包和校验该目录。

@hqhq1025

hqhq1025 commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Maintainer direction: please base the production version of this work on #6.

The native Windows executor belongs in this repository, but it should implement the same maka.cu/2 semantic contract as the macOS executor. We should not retain maka.cu.windows/0 as a second production protocol or keep two native adapter boundaries.

The C#/.NET implementation remains useful as comparative executor evidence. The planned Rust/direct-COM prototype should be evaluated behind the same contract rather than creating another product integration path. The implementation choice should be made from measured results in:

  • UIA identity fidelity and deterministic stale/replaced-control refusal
  • semantic action coverage and verification
  • target-window WGC behavior under occlusion
  • cancellation and hung-provider recovery
  • parent-death cleanup and residual-process checks
  • clean-machine deployment, startup/first-frame latency, memory, and complete distribution size
  • real tasks across Win32, WPF, WinUI/UWP, Electron/Chromium, and a problematic legacy provider

Required production constraints:

  1. Advertise only the shared semantic capabilities.
  2. Do not accept model-provided coordinates.
  3. Do not add semantic-to-pixel, foreground, global-input, or automatic replay fallbacks.
  4. Native events may be used internally only for an already PID/HWND- and observation-bound semantic action.
  5. Keep distributionReady: false until provenance, servicing, clean-machine, and packaged Maka acceptance are complete.

Please retain the exact local validation evidence in the PR description: OS/build, applications and tasks, commands, passed and skipped checks, performance/package measurements, and known unsupported cases.

Companion foundations:

After those land, this PR can either be rebased and converted to the shared contract or remain a clearly labeled comparative prototype while the chosen executor implementation is prepared.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants