Skip to content

feat(computer-use): integrate Windows native backend - #4409

Open
sunheyi6 wants to merge 5 commits into
apache:mainfrom
sunheyi6:codex/windows-cu-host-integration
Open

feat(computer-use): integrate Windows native backend#4409
sunheyi6 wants to merge 5 commits into
apache:mainfrom
sunheyi6:codex/windows-cu-host-integration

Conversation

@sunheyi6

@sunheyi6 sunheyi6 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Why

Windows builds of Maka currently have no Computer Use backend. Issue #4318 established the direction, and the native helper now has a concrete implementation in maka-agent/maka-cu#5. Maka still needs to supervise that helper, adapt its private protocol to Runtime's Computer Use contracts, select it on Windows, and package it only when provenance is ready.

What changes

  • add a dedicated WindowsCuService supervisor for maka.cu.windows/0
  • verify the helper SHA-256 before spawn and enforce initialize, timeout, cancel-grace, restart, and generation semantics
  • invalidate every observation from a helper generation after exit or restart
  • preserve outcome_unknown when a delivered mutation loses its helper
  • adapt window enumeration, UIA observations, WGC screenshots, set_value, and click_element to Runtime Computer Use contracts
  • require an explicit or uniquely resolved target and fail closed on ambiguity
  • select windows-native on Windows without enabling global coordinate or keyboard input fallbacks
  • resolve a windowsCu manifest entry in Desktop, gate packaged use on distributionReady: true, and conditionally stage the helper
  • add a local preparation script that builds the sibling maka-cu source, copies the helper, and pins its digest
  • document the development and packaged-support boundary

Validation

  • @maka/computer-use typecheck and build
  • focused backend, selector, and supervisor tests: 7/7
  • Desktop Computer Use host tests: 5/5
  • Desktop main-process build
  • Biome checks for all affected source files
  • ASF license-header audit
  • real published .NET helper + WinForms fixture integration: 2/2
    • initialize, preflight, and window enumeration
    • observe with WGC screenshot → set_value → re-observe and verify the updated value
  • no helper or fixture process remained after the integration run

The helper artifact remains distribution-ineligible by default. This draft makes Windows Computer Use usable from source after running the preparation command; release packaging should be enabled only after the companion helper PR and artifact provenance are accepted.

Refs #4318
Native helper dependency: maka-agent/maka-cu#5
Related hardening: #3785

中文说明

此 PR 把 Windows 原生 helper 接入 Maka:新增独立进程监管器、Runtime backend、Windows 平台选择、Desktop 清单和打包门禁,并保留一次性快照、全会话失效和 outcome_unknown 等安全语义。

真实联调已经跑通:Maka Host 启动发布版 .NET helper,观察 WinForms 窗口并通过 WGC 获取截图,执行 set_value,随后重新观察确认值已更新。

当前可以从源码使用;正式安装包仍需等待 helper 上游 PR、产物来源和分发门禁确认后再把 distributionReady 设为 true

关联 issue:#4318。原生 helper PR:maka-agent/maka-cu#5

@sunheyi6

sunheyi6 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

@liugddx @M4n5ter — this is the product-side implementation following the bounded Windows direction discussed in #3785 and #4318. The native dependency is maka-agent/maka-cu#5.

Could you review the dedicated private-protocol supervisor, helper-generation/session invalidation, outcome_unknown handling, and the distributionReady gate? The real published helper + fixture path passes observe/WGC screenshot → set_value → fresh readback. The PR remains Draft because clean-machine measurements, accepted artifact provenance, and packaged conversation-level acceptance are still open.

中文说明

这是 #3785 / #4318 受限 Windows 方案的产品接入 PR,原生依赖为 maka-agent/maka-cu#5

请重点审阅私有协议监管器、helper 代次与全会话失效、outcome_unknown 处理以及 distributionReady 门禁。发布版 helper 的真实联调已经通过观察/WGC 截图、填值和新观察回读;干净机器测量、正式产物来源和打包后的对话级验收仍未完成,所以保持 Draft。

@sunheyi6

sunheyi6 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Local xAI/Grok testing found and fixed a provider-schema compatibility issue before helper dispatch. Zod tuples emitted JSON Schema items: [...], which xAI rejects. Provider-facing geometry fields now use one item schema plus exact minItems/maxItems; execution-side tuple semantics remain unchanged.

The Windows selector also narrows the model-facing action enum to the helper's actual surface: list_apps, observe, screenshot, click_element, set_value, and wait. Raw coordinate/key/scroll actions are no longer advertised on Windows.

Validation: focused Runtime wire/tool tests and Windows selector tests pass; Runtime, Computer Use, and Desktop main builds pass. Commit: bc721ae.

中文说明

本地 xAI/Grok 测试发现并修复了 helper 执行前的工具 schema 兼容问题。坐标 tuple 之前会生成 xAI 不接受的 items: [...];现在改为单一 items schema 加精确长度限制,执行层 tuple 语义不变。

Windows 侧同时只向模型暴露 helper 当前支持的六个动作,不再声明坐标、键盘和滚动操作。相关测试与构建均通过。

@sunheyi6

sunheyi6 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

A second local acceptance failure exposed an artifact-closure bug. The earlier preparation script copied only maka-cu-windows.exe. The 151 KB apphost exited immediately; even the 180 MB managed single-file executable still requires five Windows Desktop native companions. Without them, initialize can succeed but the first UIA call fails in System.Windows.Automation.CacheRequest, after which the Host exhausts its restart budget.

Commit 48d6a33 fixes this end to end:

  • publish with the native self-contained/single-file contract
  • reject the small framework-dependent apphost and incomplete artifacts
  • copy the executable plus all required native companions as one directory
  • record SHA-256 and size for every file
  • verify the exact directory closure before Host selection
  • package the directory rather than one exe
  • fail closed for missing, extra, or tampered companion files
  • ensure the real integration test disposes the backend on failure

The corrected Desktop artifact passes real helper startup/window enumeration and the full fixture observe/WGC screenshot → set_value → fresh readback integration.

中文说明

第二次本地验收发现了产物闭包问题:之前脚本只复制 exe。151 KB apphost 会立即退出;即使 180 MB 的 managed single-file exe,也仍需要五个 Windows Desktop 原生 DLL。缺少这些 DLL 时握手可能成功,但第一次 UIA 调用会失败,最终表现为 helper 重启预算耗尽。

提交 48d6a33 已改为整体发布、复制、逐文件哈希校验和目录打包,并对缺失、多余或篡改文件 fail closed。修正后的 Desktop 产物已经通过真实 helper 枚举和完整 fixture 操作联调。

@liugddx
liugddx requested a review from hqhq1025 September 1, 2026 02:21
@sunheyi6
sunheyi6 marked this pull request as ready for review September 1, 2026 02:52
@hqhq1025

hqhq1025 commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Maintainer direction: this PR should be rebased and reworked after #4497.

The current revision is not ready to merge because it creates Windows-specific product authority that should be shared:

  • WindowsCuService duplicates generic child-process supervision already owned by the Computer Use host layer.
  • maka.cu.windows/0 creates a second production executor contract beside maka.cu/2.
  • Windows-specific model-schema narrowing makes the model surface platform-dependent.

The target structure is:

Maka Runtime
  -> shared supervised-child RPC/lifecycle core
  -> thin Windows CuDispatchBackend adapter
  -> Windows maka-cu executor speaking maka.cu/2

Please update this PR as follows:

  1. Rebase after refactor(computer-use): enforce semantic-only actions and enable guarded keyboard input #4497 and remove the Windows-only action enum/schema override. The global schema will already exclude coordinate mutation.
  2. Reuse or extract the shared supervised-child lifecycle core for spawn, handshake, request tracking, cancellation, deadlines, restart/backoff, generation invalidation, and disposal. Keep only Windows-specific process and artifact details in the platform layer.
  3. Consume the shared maka.cu/2 contract from maka-cu; do not retain a separate production protocol adapter.
  4. Keep the existing semantic guarantees: exact PID/HWND authority, stale/replaced target refusal, one-use observations, verified outcomes, and no replay after outcome_unknown.
  5. Keep distributionReady: false until clean-machine, provenance, servicing, packaged conversation E2E, and the real-application usability matrix are complete.

The .NET helper integration and tests are valuable evidence. They can be preserved while the helper is adapted to the shared executor contract. Please also update the PR description with the exact real-app matrix, commands, environment, unsupported actions, and skipped checks for the submitted revision.

Foundation PRs:

This is a request to converge the implementation on the maintained architecture, not a rejection of the Windows executor work itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/XL Under 2500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants