feat(windows): add native Computer Use helper - #5
Conversation
|
@liugddx @M4n5ter — this is the proposed long-term native code home from apache/maka#4318, paired with apache/maka#4409. Please review whether targeting 中文说明这是 apache/maka#4318 建议的原生代码长期归属,并与 apache/maka#4409 配套。 请确认目标分支 |
|
Local Desktop acceptance clarified the publish contract: PublishSingleFile contains the managed payload, but Windows Desktop/WPF still emits five required native companion DLLs. Copying only the exe can complete initialize and then fail when UI Automation initializes. Commit 96351b8 updates the native manifest to hash and size the complete publish closure and documents the required companions. The paired Host PR now copies, packages, and verifies that exact directory. 中文说明本地 Desktop 验收确认:PublishSingleFile 只表示托管 payload 单文件,Windows Desktop/WPF 仍会输出五个必需原生 DLL。只复制 exe 可能握手成功,但在 UIA 初始化时失败。 提交 96351b8 已让原生 manifest 记录整个发布闭包,并明确 companion DLL 要求;配套 Host PR 会整体复制、打包和校验该目录。 |
|
Maintainer direction: please base the production version of this work on #6. The native Windows executor belongs in this repository, but it should implement the same The C#/.NET implementation remains useful as comparative executor evidence. The planned Rust/direct-COM prototype should be evaluated behind the same contract rather than creating another product integration path. The implementation choice should be made from measured results in:
Required production constraints:
Please retain the exact local validation evidence in the PR description: OS/build, applications and tasks, commands, passed and skipped checks, performance/package measurements, and known unsupported cases. Companion foundations:
After those land, this PR can either be rebased and converted to the shared contract or remain a clearly labeled comparative prototype while the chosen executor implementation is prepared. |
Why
Apache Maka issue #4318 needs a native Windows Computer Use executor with explicit window authority, bounded lifecycle behavior, and evidence that does not depend on global input or screen-rectangle capture.
What changes
maka.cu.windows/0JSON-RPC protocolset_valueandclick_elementthrough UIA patternsCreateForWindowdistributionReady: falsepending release ownershipThis PR intentionally keeps the existing Go CLI/MCP implementation unchanged. The new helper is a separate supervised product process.
Validation
dotnet buildfor helper and WinForms fixture: 0 warnings, 0 errorsgo test ./...set_value, and re-observe verificationRelated: apache/maka#4318
Companion host integration: apache/maka#4409.
中文说明
此 PR 在
maka-cu中加入 Windows 原生 Computer Use helper:使用 UI Automation 观察和执行语义操作,使用 Windows Graphics Capture 对指定窗口截图,并通过一次性快照、窗口身份校验、取消、父进程退出和 helper 重启机制避免错误操作与静默重试。现有 Go CLI/MCP 实现保持不变。发布产物目前仍标记为
distributionReady: false,等待上游确定正式构建、签名和分发责任。关联 Apache Maka issue:apache/maka#4318。