feat(computer-use): add Windows .NET feasibility spike - #4346
Conversation
Add a supervised UIA/WGC helper, deterministic Windows lifecycle fixtures, reproduction scripts, and published-artifact results for the bounded spike. Keep product integration disabled and record outstanding feasibility evidence. Cover C# and project files in the existing ASF source-header audit. Refs apache#4318 Refs apache#3785 Generated-by: OpenAI Codex
|
Codex-assisted update, posted on behalf of @sunheyi6. @liugddx @M4n5ter — the draft feasibility prototype is ready for an initial look, following your C#/.NET spike direction and bounded semantic preview / safety guidance. The published local artifact passes the 34 lifecycle and 3 protocol checks. This is still an incomplete feasibility submission: clean-machine execution, same-window control replacement, and packaging measurements remain explicit TODOs. It does not enable the product backend or close #4318. Could you review the private helper/TypeScript supervision boundary, target identity and cancellation semantics, and whether the proposed path from #4318 to the #3785 semantic preview matches the intended scope? The follow-up implementation plan is recorded in this #4318 comment. An independent human review is requested; the automated verification is not a substitute. |
|
The feasibility work has now moved into the proposed two-repository implementation:
This PR remains useful as the original in-tree evidence checkpoint. The active implementation should be reviewed in the two linked Draft PRs to avoid maintaining duplicate helper implementations. 中文说明实验工作已经迁移为双仓库实现:原生 helper 在 maka-agent/maka-cu#5,Maka Host 接入在 #4409;最新计划和剩余验收门禁已更新到 #4318。 本 PR 保留为最初的主仓库实验记录;后续实现审阅应放在两个新 Draft PR,避免长期维护重复 helper。 |
Summary
Refs #4318. Part of the Windows Computer Use epic #3785 and roadmap #2142.
Windows Computer Use needs evidence that a long-lived C#/.NET helper can safely support UIA, target-window capture, and recovery before product integration. This draft adds the bounded feasibility prototype requested in #3785, under
experiments/maka-cu-windows/.CreateForWindowcapture, plus deterministic fixtures and drivers for occlusion, hung providers, restart, stale tokens, and abrupt parent exit.win-x64publishing, artifact hashes, reproduction commands, and D1–D6 recommendations inREADME.md/RESULTS.md.This does not enable Windows Computer Use in Maka. It leaves the macOS executor, public protocol, approval flow, and bundled-tool pins unchanged. No global input or screen-rectangle fallback is added. The executable is unsigned with
distributionReady: false.Verification
Remaining feasibility work
Signing/distribution readiness and production host integration are follow-up work, not additional conditions for an unsigned spike. This draft intentionally uses
Refs, not a closing keyword, because #4318 is not complete.AI use
Tool(s) and scope: OpenAI Codex; implementation used
gpt-5.6-lunawith high reasoning at the contributor's request. Codex also assisted with review, local verification, documentation, and PR preparation. Human contributor of record: @sunheyi6. The commit includesGenerated-by: OpenAI Codex; an independent human review is still required.Checklist
Lint, format and the experiment/header suites passed; the aggregate box stays unchecked because full workspace typechecking was not run.
Does this PR entail a change in behavior?
Only the new experiment and source-header audit gain behavior; the shipping application does not.
中文说明
本 Draft PR 关联 #4318、Windows Computer Use 总 issue #3785 和路线图 #2142,提交维护者要求的 C#/.NET 可行性实验,不代表 Windows 功能已经接入 Maka。
内容包括常驻 helper、私有 stdio 协议、MTA UIA、快照和 opaque token、语义操作及结果回读、真实目标窗口 WGC 截图、取消与卡死恢复,以及测试窗口、复现脚本、自包含打包和 D1–D6 决策记录。新增 C# 文件类型也纳入 ASF 许可证审计;不改 macOS、公开协议、现有授权流程或 bundled-tool pins,不引入全局输入或屏幕矩形截图降级。
本机 Windows 11 Insider 上,打包产物生命周期 34 项、协议回归 3 项、许可证测试 43 项均通过,全仓 lint、格式和许可证审计通过。没有运行全仓构建、类型检查、测试和 desktop/UI Knip;本机实验也不能替代干净机器或 Maka 安装包端到端验证。
仍需补齐同窗口控件替换与身份复用分析、无 .NET/SDK 的干净交互式 Windows 环境、稳定版本证据,以及启动/首帧/内存/原生库解压等测量,并评估 .NET 10 基线。因此保持 Draft,不自动关闭 #4318。签名属于后续正式分发工作,不额外作为无签名实验的前置门槛。
实现按贡献者要求使用 Codex 的 luna 高思考模式;Codex 也参与审查、验证、文档和 PR 准备。人工贡献者为 @sunheyi6,仍需维护者独立人工审查。