vision-framework-skill

An agent skill for on-device image analysis using Apple's Vision Framework — OCR, face detection, image classification, barcode reading, and body pose estimation, all running locally on your Mac.

Compatible with any agent framework that supports the Agent Skills specification — Claude Code, OpenClaw, Hermes, Codex, Antigravity, Cursor, Windsurf, and more.

Zero network calls, zero API keys — everything runs locally on your Mac.

Features

Feature	Description
ocr	Text recognition with confidence scores and bounding boxes (100+ languages)
classify	Image classification into 1000+ categories
faces	Face detection with bounding boxes
humans	Human body detection with bounding boxes
animals	Cat/dog detection with species labels
barcodes	QR code and barcode reading (any symbology)
contours	Edge/contour detection
featureprint	Image feature vector for similarity comparison
pose	Human body pose with joint landmarks
all	Run all of the above

Requirements

• macOS 13+
• Swift toolchain (Xcode or Command Line Tools: xcode-select --install)

Installation

🤖 Via AI Agent (Recommended)

Tell any AI coding agent:

"Install the Vision Framework skill from haoyiyin/vision-framework-skill"

Your agent will clone, build the Swift CLI tool, and configure everything automatically. Works with Pi, Claude Code, Cursor, Codex, Copilot, and any agent supporting the Agent Skills specification.

💡 Requires macOS 13+ and Xcode Command Line Tools (xcode-select --install). First-time build compiles the Swift CLI (~30 seconds).

🛠️ Manual Installation

1. Clone the skill

git clone https://github.com/haoyiyin/vision-framework-skill.git ~/.agents/skills/vision-framework

2. Symlink into your agent's skill directory

For Claude Code:

ln -s ~/.agents/skills/vision-framework ~/.claude/skills/vision-framework

3. Build the CLI tool

cd ~/.agents/skills/vision-framework/swift && swift build -c release

Usage

Once installed, your agent will automatically discover and invoke the skill when you ask it to analyze images:

• "What text is in this screenshot?"
• "Detect faces in this photo"
• "Read the QR code in this image"
• "What animal is in this picture?"
• "Analyze this image and tell me everything you see"

CLI Usage

You can also use the binary directly without any agent:

# OCR
VisionCLI screenshot.png --features ocr

# Face detection with confidence threshold
VisionCLI photo.jpg --features faces --confidence 0.8

# Multiple features
VisionCLI photo.jpg --features ocr,faces,classify

# Japanese OCR
VisionCLI document.png --features ocr --language ja-JP

# Multiple images
VisionCLI img1.jpg img2.jpg --features all

# Limit results
VisionCLI photo.jpg --features classify --top 5

Options

Option	Default	Description
--language <tag>	en-US	OCR language hint (BCP 47)
--recognition-level <fast|accurate>	accurate	OCR speed vs quality
--confidence <0.0-1.0>	0.0	Minimum confidence threshold
--top <n>	unlimited	Max results per feature

Output Format

JSON to stdout. Bounding boxes use normalized coordinates (0.0–1.0).

{
  "imagePath": "/path/to/image.jpg",
  "imageSize": {"width": 1920, "height": 1080},
  "features": {
    "ocr": [
      {"text": "Hello World", "confidence": 1.0, "boundingBox": {"x": 0.1, "y": 0.2, "width": 0.3, "height": 0.05}}
    ],
    "classify": [
      {"label": "document", "confidence": 0.56}
    ],
    "faces": [],
    "humans": [],
    "animals": [],
    "barcodes": [],
    "contours": null,
    "featureprint": null,
    "pose": []
  },
  "errors": []
}

How It Works

The skill bundles a Swift CLI tool (VisionCLI) that wraps Apple's Vision Framework. Your agent:

1. Builds the CLI tool (first time only)
2. Runs VisionCLI <image> --features <features>
3. Parses the JSON output and presents the results

Vision Framework runs on the Apple Neural Engine (on Apple Silicon), so inference is fast and efficient — no GPU setup, no model downloads.

License

MIT