Skip to content

feat(computer-use): add Windows .NET feasibility spike - #4346

Draft
sunheyi6 wants to merge 1 commit into
apache:mainfrom
sunheyi6:codex/windows-cu-feasibility-spike
Draft

feat(computer-use): add Windows .NET feasibility spike#4346
sunheyi6 wants to merge 1 commit into
apache:mainfrom
sunheyi6:codex/windows-cu-feasibility-spike

Conversation

@sunheyi6

Copy link
Copy Markdown
Contributor

Summary

Refs #4318. Part of the Windows Computer Use epic #3785 and roadmap #2142.

Windows Computer Use needs evidence that a long-lived C#/.NET helper can safely support UIA, target-window capture, and recovery before product integration. This draft adds the bounded feasibility prototype requested in #3785, under experiments/maka-cu-windows/.

  • Private stdio JSON-RPC, a dedicated MTA UIA lane, snapshot-bound opaque tokens, semantic mutation/readback, and bounded cancellation/shutdown.
  • Real WGC CreateForWindow capture, plus deterministic fixtures and drivers for occlusion, hung providers, restart, stale tokens, and abrupt parent exit.
  • Self-contained win-x64 publishing, artifact hashes, reproduction commands, and D1–D6 recommendations in README.md / RESULTS.md.
  • Extend the ASF header audit to cover the new C# source and project file types; no exclusion bypass.

This does not enable Windows Computer Use in Maka. It leaves the macOS executor, public protocol, approval flow, and bundled-tool pins unchanged. No global input or screen-rectangle fallback is added. The executable is unsigned with distributionReady: false.

Verification

  • Published-artifact lifecycle suite: 34 PASS, 0 failures on an interactive Windows 11 Insider x64 desktop (10.0.26220).
  • Protocol regression suite: 3 PASS, 0 failures; malformed methods, unknown cancellation IDs, and undrained stdout plus EOF.
  • WGC produced a decoded PNG at the target's DWM bounds; the 4,000-pixel fixture sentinel survived occlusion and uncovering. No fixture/helper processes remained after teardown.
  • ASF header tests: 43 passed; full source-header audit, repository Biome lint and format checks passed.
  • Full workspace build/typecheck/test and desktop/UI Knip were not run; this is an isolated .NET experiment with no TypeScript product changes. These results are not packaged Maka E2E or clean-machine evidence.

Remaining feasibility work

  • Exercise same-window control replacement and complete the identity/reuse analysis.
  • Run the declared published layout on an interactive clean Windows machine with no .NET runtime or SDK; record supported-release evidence separately from this Insider run.
  • Measure cold/warm startup, first-frame latency, memory, distribution/extraction footprint and permissions; evaluate .NET 10 as the forward baseline (.NET 8 is the installed local spike SDK).
  • Finish the feat(windows): feasibility spike for a supervised C#/.NET maka-cu Windows helper #4318 decision record from that evidence before enabling the production preview.

Signing/distribution readiness and production host integration are follow-up work, not additional conditions for an unsigned spike. This draft intentionally uses Refs, not a closing keyword, because #4318 is not complete.

AI use

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: OpenAI Codex; implementation used gpt-5.6-luna with high reasoning at the contributor's request. Codex also assisted with review, local verification, documentation, and PR preparation. Human contributor of record: @sunheyi6. The commit includes Generated-by: OpenAI Codex; an independent human review is still required.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Lint, format and the experiment/header suites passed; the aggregate box stays unchecked because full workspace typechecking was not run.

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

Only the new experiment and source-header audit gain behavior; the shipping application does not.

中文说明

本 Draft PR 关联 #4318、Windows Computer Use 总 issue #3785 和路线图 #2142,提交维护者要求的 C#/.NET 可行性实验,不代表 Windows 功能已经接入 Maka。

内容包括常驻 helper、私有 stdio 协议、MTA UIA、快照和 opaque token、语义操作及结果回读、真实目标窗口 WGC 截图、取消与卡死恢复,以及测试窗口、复现脚本、自包含打包和 D1–D6 决策记录。新增 C# 文件类型也纳入 ASF 许可证审计;不改 macOS、公开协议、现有授权流程或 bundled-tool pins,不引入全局输入或屏幕矩形截图降级。

本机 Windows 11 Insider 上,打包产物生命周期 34 项、协议回归 3 项、许可证测试 43 项均通过,全仓 lint、格式和许可证审计通过。没有运行全仓构建、类型检查、测试和 desktop/UI Knip;本机实验也不能替代干净机器或 Maka 安装包端到端验证。

仍需补齐同窗口控件替换与身份复用分析、无 .NET/SDK 的干净交互式 Windows 环境、稳定版本证据,以及启动/首帧/内存/原生库解压等测量,并评估 .NET 10 基线。因此保持 Draft,不自动关闭 #4318。签名属于后续正式分发工作,不额外作为无签名实验的前置门槛。

实现按贡献者要求使用 Codex 的 luna 高思考模式;Codex 也参与审查、验证、文档和 PR 准备。人工贡献者为 @sunheyi6,仍需维护者独立人工审查。

Add a supervised UIA/WGC helper, deterministic Windows lifecycle fixtures,
reproduction scripts, and published-artifact results for the bounded spike.
Keep product integration disabled and record outstanding feasibility evidence.
Cover C# and project files in the existing ASF source-header audit.

Refs apache#4318
Refs apache#3785
Generated-by: OpenAI Codex
@sunheyi6

Copy link
Copy Markdown
Contributor Author

Codex-assisted update, posted on behalf of @sunheyi6.

@liugddx @M4n5ter — the draft feasibility prototype is ready for an initial look, following your C#/.NET spike direction and bounded semantic preview / safety guidance.

The published local artifact passes the 34 lifecycle and 3 protocol checks. This is still an incomplete feasibility submission: clean-machine execution, same-window control replacement, and packaging measurements remain explicit TODOs. It does not enable the product backend or close #4318.

Could you review the private helper/TypeScript supervision boundary, target identity and cancellation semantics, and whether the proposed path from #4318 to the #3785 semantic preview matches the intended scope? The follow-up implementation plan is recorded in this #4318 comment. An independent human review is requested; the automated verification is not a substitute.

中文说明

此消息由 Codex 协助、代表 @sunheyi6 发布。

根据两位此前对 C#/.NET 实验和受限语义首版安全边界的建议,现已提交 Draft 供初步审阅。打包产物在本机通过 34 项生命周期和 3 项协议检查,但干净机器、同窗口控件替换和打包测量仍待补齐;没有启用正式后端,也不关闭 #4318

请重点看看私有 helper 与 TypeScript 监管的边界、目标身份和取消语义,以及从 #4318 走向 #3785 首版的实施方向是否符合预期。后续计划已写入 #4318;自动验证不代替独立人工审查。

@sunheyi6

sunheyi6 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

The feasibility work has now moved into the proposed two-repository implementation:

This PR remains useful as the original in-tree evidence checkpoint. The active implementation should be reviewed in the two linked Draft PRs to avoid maintaining duplicate helper implementations.

中文说明

实验工作已经迁移为双仓库实现:原生 helper 在 maka-agent/maka-cu#5,Maka Host 接入在 #4409;最新计划和剩余验收门禁已更新到 #4318

本 PR 保留为最初的主仓库实验记录;后续实现审阅应放在两个新 Draft PR,避免长期维护重复 helper。

@github-actions github-actions Bot added effort/XXL Over 2500 readable lines and removed effort/XL Under 2500 readable lines labels Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/XXL Over 2500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant