[bobleer]

Summary

ComputerUse spent extra model turns on setup, repeated observations and individual inputs, while every app observation included a screenshot. This change makes the prompt and compact tool schema prefer one batch of already-decided inputs followed by one observation, and adds optional include_image: false for flows whose AX text is sufficient.

Existing calls still include images by default. Canvas and visual decisions retain image-bearing observations. Stop, cancellation, target freshness, partial receipts and authorization remain enforced. macOS focus-and-type/shortcut operations also avoid redundant nested desktop witnesses while retaining the outer operation witness and native input checks.

Type and Areas

Type: Performance improvement, tests, documentation.

Areas: Agent prompts, Core ComputerUse tool/host port, Desktop native observation, macOS fixtures.

Motivation / Impact

• Reuse known app selectors and post-input observations; batch predictable search, keyboard and form steps up to the next decision boundary.
• Remove conflicting legacy guidance that encouraged global input, script workarounds, digest-based verification and duplicate reads. The ComputerUse prompt shrinks from 17,177 to 10,701 bytes (37.7%); user-selected reasoning effort is unchanged.
• Let the Desktop adapter skip model frame retrieval/encoding for explicit AX-only observations while refreshing node identities. The additive host method preserves legacy provider behavior and existing image defaults.
• Report observation time separately and provide a reproducible four-case native latency fixture.

Focused REA analysis of shipped Codex CUA JavaScript established grouped model-facing calls and separately selected observations; no server-side prompt or Codex runtime speed comparison was recovered. Findings, limits and acceptance evidence are recorded in docs/architecture/computer-use-control.md. No third-party implementation or research artifacts are included.

Verification

Upstream CI run 38347581371 and the repository object-size workflow passed for commit 3518977089bb85a65ba2aa0c795b1b3fc037d78a: 11 successful checks, 2 impact-selected jobs skipped. Windows desktop lib tests encountered the repository's existing Tauri #13419 loader exception (0xc0000139, before any Rust test ran); the configured probe accepted only that known signature. Windows compilation, CLI and the separate contract suites passed. This PR does not alter that exception or claim Windows native desktop test execution.

AI-assisted; focused unit/contracts and local macOS native behavior were fully exercised. Windows/Linux native behavior and remote end-to-end scenarios remain untested locally and are not implied by these results.

Passed after synchronizing with upstream e24de2926:

• cargo test --locked -p openbitfun-core --no-default-features --features agent-runtime,git,tools-computer-use --lib computer_use_tool::tests — 53 tests.
• cargo test --locked -p openbitfun-agent-content --test prompt_catalog_contracts — 8 tests.
• cargo test --locked -p openbitfun-agent-runtime --no-default-features --features agent-runtime --test agent_definition_contracts prompt_contracts — 10 tests.
• cargo test --locked -p openbitfun-core --no-default-features --features agent-runtime,git,tools-computer-use --lib prompt_builder_impl::tests::runtime_context — 7 tests.
• pnpm run fmt:rs, node --check scripts/test-macos-input-controls.mjs, git diff --check, and the introduced Git object size check passed.

Native acceptance on the implementation before that unrelated SSH-only upstream synchronization:

• cargo test --locked -p openbitfun-desktop --features devtools --test computer_use_native_roundtrip --no-run — built successfully.
• node scripts/test-macos-input-controls.mjs --orchestration — strict production Tool → host → native acceptance passed, including AX-only fresh/stale node identity, default-image recovery, exact Unicode input, semantic/image targets, scroll, canvas batches, preview/revocation and Stop. Foreground, cursor and target/window isolation assertions remained enabled.
• node scripts/test-macos-input-controls.mjs --latency — three repetitions, 12 fresh cases, all passed. Locked/concurrently active desktop attempts were treated as inconclusive and excluded.

Same updated development build, identical five native inputs, 12-node AX tree:

Strategy	Tool calls	Returned observations / images	Median host time (3 samples)
Single inputs, default images	5	5 / 5	1,749 ms
One batch, default image	1	1 / 1	464 ms
Single inputs, AX only	5	5 / 0	176 ms
One batch, AX only	1	1 / 0	149 ms

These compare strategies in the same build; batching already existed. They do not measure model inference, overall task completion time, large application trees, Codex runtime or a before/after speedup. Each result must expose the exact final text and confirmed shortcut delivery before recovery reads or waits. Setup, initial observation and cleanup are excluded; the separately timed immediate fixture check adds 0.073–0.130 ms.

Reviewer Notes

No new Tauri command, persisted required field, dependency or remote capability. Remote workspace/headless no-local-fallback contracts passed. Remote workspace, remote control, Peer Device Mode and Detached Dispatch were not exercised end to end. The macOS native fixture requires an unlocked, idle desktop and never unlocks it automatically.

Checklist

• This PR is focused and does not include secrets, temporary prompts, generated scratch files, or unrelated artifacts.
• Relevant verification is recorded above, or skipped checks are explained.
• User-facing strings, docs, and locales are updated where applicable.