feat: ui-helper (windows/displays/permissions/captureScreen) + extended Vision API - #9
Merged
Merged
Conversation
- src/native/ui-helper.swift: read-only CoreGraphics introspection helper (CGGetOnlineDisplayList so asleep displays stay visible; no AppKit) - scripts: ui-helper joins vision-helper/pdf-helper in the prebuilt pipeline - src/ui.ts: listWindows / listDisplays / checkPermissions / captureScreen (screencapture-based, returns paths + geometry, never image bytes) - README: UI API section with privacy invariant (eyes, not hands) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nd pixel ops vision-helper (macOS 13 target, newer features behind #available): - OCR options: --lang, --auto-lang, --no-correction, --custom-words, --fast, --roi (results mapped back to full-image space), --min-text-height - --document-structure (macOS 26, RecognizeDocumentsRequest): title, paragraphs, tables with spans, lists, barcodes, detected data with positions - --entities (NSDataDetector over stdin), --text-rects, --compare (feature print distance), --image-info, --languages, --capabilities - people/scenes: --face-landmarks (+capture quality, head pose), --humans, --body-pose, --hand-pose, --animals, --animal-pose, --horizon, --contours, --saliency attention|objectness (+heatmap), --aesthetics (15+), --smudge (26+, reports supported:false when the model is absent on this hardware) - pixel ops returning paths: --crop, --document-crop (CIPerspectiveCorrection), --foreground-mask (14+), --person-mask TypeScript: - src/vision.ts: typed wrappers for everything above, UnsupportedOnThisMacOSError - ocr(): TextRecognitionOptions, opt-in content-hash cache, onProgress for PDFs - captureScreen(): sha256 of the PNG - CLI: OCR tuning flags, --structure/--entities/--text-regions/--humans/ --face-landmarks/--saliency/--aesthetics/--info/--capabilities/--languages - tests: test/vision.test.ts (22 cases); README; changeset (minor); node >= 20 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ft/TS boilerplate
- src/helper.ts: one execHelper/runHelper/runGated for all three binaries
(timeouts, buffer, exit-code-2 → UnsupportedOnThisMacOSError), tmpOutPath,
sha256/fileSha256 — replaces three divergent runners
- vision-helper: fd 1 diverted to stderr process-wide, JSON written to the saved
stdout (one payload, no per-mode dup2 hack); perform()/fail() replace 19
copy-pasted catch blocks; Box {x,y,width,height,confidence} unifies
Face/TextRect/Human/DocRegion so TS needs no RawBox/DocBox mapping layer;
supportedLanguages(), macOS15/26 flags, hoisted ISO8601 formatter
- ocr cache key = content hash + canonical textOptionArgs (order-independent);
onProgress moved to OcrOptions; ocrPdf typed as Omit<OcrOptions,'format'>
- captureScreen reads the PNG once (IHDR + sha256 from one buffer)
- CLI: top-level await early exit for --capabilities/--languages (one spawn),
single memoized OCR pass shared by --ocr/--entities, CLASSIC_FLAGS table
- visionCapabilities() memoized per process; ui-helper hoists --all check
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… helper errors Found by an end-to-end pass over the built package: - detectContours still mapped the pre-refactor c.w/c.h wire names, so every contour came back with width/height undefined. Returns the helper shape now, with a regression test asserting width/height on every rect-shaped result. - vision-helper referenced symbols introduced in macOS 14/15/26 SDKs unconditionally, so the swiftc fallback in install-native.js could not build on an older Mac (#available guards runtime, not the SDK). Modern paths are now behind -DSDK_14/15/26, which both build scripts derive from "xcrun --sdk macosx --show-sdk-version"; a build without them still compiles, reports the feature as false in --capabilities and exits 2, which maps to UnsupportedOnThisMacOSError. Verified by compiling all four SDK tiers. - Helper failures leaked Node's "Command failed: /path/to/vision-helper --flags" plus a stack trace. execHelper now surfaces the helper's own ERROR line while preserving the exit status (so gating still works) and stderr (so captureScreen can still quote screencapture). Tests: 82 unit (6 new: wire-shape, capability gating, error reporting). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review of feat/ui-helper found three blockers before release: - The branch dropped macOS 12 (native target 12→13) and Node 18 under a `minor` changeset, so a `^1.5.0` consumer on macOS 12 would auto-upgrade into binaries that cannot launch. The platform drop traced to a single un-gated `automaticallyDetectsLanguage` call (macOS 13+); wrapping it in `if #available` — the pattern the file already uses for 14/15/26 — lets all three helpers build at macos12 with every gated feature still enabled. The Node bump was unrelated to this branch (`fetch`/`AbortSignal.timeout` in ollama.ts predate it and ran fine on 18). - The suite was red: two pre-existing detectDocument tests timed out at the default 5s when the new model-bound tests ran alongside them, though they pass in isolation and the helper itself takes 0.12s. Added vitest.config.ts (30s timeout, no file parallelism, worktree copies excluded): 82/82 pass. - Release builds now refuse an SDK too old to compile in every gated feature, instead of silently shipping binaries that report them unavailable. Also: imageAesthetics uses runGated() like its siblings instead of hand-checking the exit code; the OCR cache key includes helper and macOS version so entries do not survive an upgrade that changes results; corrected the comment above unROI, which claimed the opposite of what the code does (verified empirically that Vision reports ROI-relative coordinates). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Keeps this package in step with the consumer that will eventually drop its own copy of ui-helper. On a locked Mac, window and region capture fail outright and a full-screen capture returns only the lock screen — verified against a genuinely locked session. checkPermissions now reports screenLocked so a caller can check before starting, and captureScreen consults it on failure to say plainly that retrying cannot succeed until someone unlocks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
detectLensSmudge documents that it returns `supported: false` rather than guessing, and the helper already detected hardware without the model by sniffing VisionCore's stdout notice. CI runners fail differently: the request throws Foundation._GenericObjCError.nilError, which took the fail() path and turned a "this Mac cannot do it" into a hard error — turning the suite red on GitHub while passing locally. The image has already loaded by that point, so a throw here can only mean the model is unusable on this machine. Report it the same way as the notice path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
ui-helperas the package's third prebuilt native helper and exports a screen-capture API, plus a broad extension of the Vision surface.ui-helper — the piece macos-vision-mcp is waiting on
listWindows,listDisplays,checkPermissions,captureScreen(window, region, display; returns path + geometry + sha256, never image bytes).install-native.js/build-native-cross.js), so consumers do not need Xcode at runtime.checkPermissionsreportsscreenLocked; capture failures consult it, so a locked Mac is diagnosed outright instead of guessed at.Extended Vision API
recognizeDocument(macOS 26 — native paragraphs/tables/lists/detected data),extractEntities,compareImages,detectTextRegions,imageInfo,visionCapabilities, pose/face/animal detection, saliency, contours, aesthetics, lens smudge, and pixel ops that return paths (cropImage,cropDocument,extractForeground,personMask). OCR gains language, ROI, custom-word and correction options plus an opt-in content-hash cache.Compatibility
No breaking changes. macOS 12 and Node 18 support are retained: modern APIs are gated twice, at compile time on the SDK (
-DSDK_14/15/26) and at runtime on#available. Release builds now refuse an SDK too old to compile in every gated feature, so a CI image change cannot silently strip them.Verification
82/82 tests pass.
vitest.config.tsadded — the suite was flaky because model-bound tests ran under the default 5s timeout.🤖 Generated with Claude Code