diff --git a/.agents/skills/local-ocr/SKILL.md b/.agents/skills/local-ocr/SKILL.md index 3ca6c5c..9beb21b 100644 --- a/.agents/skills/local-ocr/SKILL.md +++ b/.agents/skills/local-ocr/SKILL.md @@ -1,70 +1,110 @@ -# light-ocr: Local OCR Skill +--- +name: local-ocr +description: Extract text from local images (PNG, JPEG) with precise coordinates, confidence scores, and stable error handling using the light-ocr CLI. Use when needing to read small/dense text in screenshots, receipts, labels, forms, or documents that a multimodal model may misread; when exact text plus bounding box coordinates are needed for field extraction, redaction, counting, or layout analysis; or when deterministic offline OCR is required without network or API calls. +--- -Use this skill when you need to extract text from local images (PNG, JPEG) with precise coordinates, confidence scores, and stable error handling — without writing Node.js integration code or relying on a multimodal model's ability to read small text. +# local-ocr -## When to use OCR instead of a multimodal model +`light-ocr` is a local OCR engine. It runs offline, returns text with coordinates and confidence, and follows a strict stdout/stderr contract for scripting. -- Small text, dense text, or text in screenshots/labels/receipts/forms that a multimodal model misreads or hallucinates. -- When you need exact text + bounding box coordinates (for field extraction, redaction, counting, or downstream layout analysis). -- When you need deterministic, offline, reproducible results (no network, no API calls). +## Scenarios -## Commands +### Screenshot with small text + +A user shares a screenshot and asks about specific text that is too small or dense to read visually. ```bash -# Full OCR: recognize text + coordinates (default action) -light-ocr image.png --format json -light-ocr image.png --format text # just text, no coordinates +# Step 1: recognize the full image +light-ocr screenshot.png --format json +``` + +If the result has low confidence or missing text in a region: -# Region-only recognition (ROI) -light-ocr recognize image.png --region 100,80,640,320 --format json +```bash +# Step 2: re-run on the specific region (coordinates from step 1 boxes) +light-ocr recognize screenshot.png --region 100,80,640,320 --format json +``` -# Detect-only: just text region boxes, no recognition -light-ocr detect image.png # output is always JSON -light-ocr detect image.png --crop # include PNG crop per box +### Form or receipt field extraction -# Diagnostics (no image read) -light-ocr info --model-info # full EngineInfo JSON -light-ocr info --version # npm/core/model triple +Need to extract specific fields (names, amounts, dates) from a form or receipt image. -# stdin -cat image.png | light-ocr recognize --stdin --type image/png --format json +```bash +# Step 1: detect where text regions are (fast, no recognition) +light-ocr detect receipt.png -# Execution provider -light-ocr recognize image.png --provider auto # default: auto-select best -light-ocr recognize image.png --provider cpu # force CPU +# Step 2: recognize only the region containing the target field +light-ocr recognize receipt.png --region 50,200,300,80 --format json ``` -## Output schema +This two-step pattern saves time on large images: detect first, then recognize only the regions of interest. + +### Counting text regions + +Need to count how many text lines or regions exist in an image. + +```bash +light-ocr detect image.png | python -c "import json,sys; print(len(json.load(sys.stdin)['pages'][0]['detections']))" +``` + +### Verifying multimodal model output + +A multimodal model claims to read text from an image. Verify the claim against deterministic OCR. + +```bash +light-ocr image.png --format text +``` + +Compare the text output with the model's claim. If they differ, trust the OCR `text` field — do not fabricate. -All `recognize`/`detect` output uses `--schema-version 1` (default). The envelope: +### Batch processing via shell + +Process multiple images sequentially with JSONL output. + +```bash +for f in *.png; do + light-ocr recognize "$f" --format jsonl +done +``` + +Each line is one page record. Check exit codes: a non-zero exit for one image does not stop the loop, but stdout for that image may be empty. + +## Decision flow + +``` +Need text from an image? +├── Know which region? → recognize --region x,y,w,h --format json +├── Need full text only? → recognize --format text +├── Need text + coordinates? → recognize --format json +├── Only need where text is? → detect +├── Large image, unsure where text is? → detect first, then recognize --region +└── Need engine info or version? → info --model-info / info --version +``` + +## Output schema ```json { "schemaVersion": 1, "source": { "kind": "image", "mediaType": "...", "identity": {}, "appliedTransforms": {} }, - "pages": [{ "index": 0, "width": ..., "height": ..., "coordinateSpace": "pageSpace", "structure": "ocr-order|detect", "lines|detections": [] }] + "pages": [{ + "index": 0, + "width": 640, "height": 480, + "coordinateSpace": "pageSpace", + "structure": "ocr-order", + "lines": [{ "id": "L0", "text": "HELLO", "confidence": 0.99, "box": [4 points] }] + }] } ``` -- `recognize`: `pages[0].lines[]` with `{ id: "L0", text, confidence, box: [4 points] }` -- `detect`: `pages[0].detections[]` with `{ id: "D0", score, box: [4 points] }` -- `--format text`: only recognized text, one line per line (no coordinates) +- `box` is 4 points in `pageSpace` (top-left origin, x right, y down, post-EXIF pixels) +- `detect` replaces `lines` with `detections[]` (`{ id, score, box }`) and sets `structure: "detect"` +- `--format text`: recognized text only, one line per line, no coordinates - `--format jsonl`: one page record per line (for streaming/batch) -## Choosing what to run - -| You want | Command | -| --- | --- | -| Full text from an image | `recognize --format text` | -| Text + coordinates | `recognize --format json` | -| Just where text is (no text) | `detect` | -| Text in a specific area | `recognize --region x,y,w,h` | -| Engine/provider info | `info --model-info` | -| Quick version check | `info --version` | - ## Exit codes -| Code | Meaning | Agent action | +| Code | Meaning | Action | | --- | --- | --- | | 0 | Success | Parse stdout | | 64 | Usage error | Fix command syntax | @@ -72,35 +112,15 @@ All `recognize`/`detect` output uses `--schema-version 1` (default). The envelop | 66 | Invalid image | Try different image | | 67 | Unsupported capability | Check `info --model-info` | | 68 | Model/bundle error | Reinstall package | -| 69 | Resource limit exceeded | Smaller image or region | +| 69 | Resource limit exceeded | Use smaller image or `--region` | | 70 | Environment/package failure | Check native addon | | 71 | Inference failure | Retry or report bug | | 72 | Internal error | Report bug | -## How to handle failures - -- **Empty result**: No text found. The image may have no text, or text is too small/low-contrast. Try `--region` on specific areas, or check with `detect` first. -- **Low confidence**: Lines with `confidence < 0.5` may be unreliable. Don't present inferred or guessed text as OCR output — always cite the actual `text` field and its `confidence`. -- **Resource limit (exit 69)**: Image too large. Use `--region` to process a sub-area, or downscale before passing. -- **Unsupported capability (exit 67)**: The requested provider or feature isn't available. Run `info --model-info` to see what's supported. - -## Important rules - -1. **Never fabricate OCR text.** Only use text from the `text` field of the result. If confidence is low, say so — don't guess. -2. **Cite coordinates when relevant.** Box coordinates are in `pageSpace` (top-left origin, x right, y down, post-EXIF pixels). Use them for field extraction, redaction, or counting. -3. **Use `--schema-version 1`** for reproducible output. Don't parse help text programmatically. -4. **Prefer `detect` first** if you only need to locate text regions (faster, no recognition). -5. **Use `--region`** to avoid processing huge images unnecessarily — detect first, then recognize specific regions. -6. **Check exit codes** before parsing stdout. Non-zero exit means stdout may be empty; stderr has the error. - -## Validation script - -```bash -# Quick smoke test: recognize a known image and check exit code -light-ocr test-image.png --format text && echo "OK: $(light-ocr test-image.png --format text | wc -l) lines" -``` - -## Related +## Rules -- [CLI design](docs/cli-design.md) — full flag reference, coordinate semantics, exit codes -- [Roadmap N1](docs/roadmap.md) — product context and acceptance criteria +1. Never fabricate OCR text. Only use the `text` field from results. If confidence < 0.5, state it. +2. Cite coordinates when relevant. Box coordinates are in `pageSpace`. +3. Use `--schema-version 1` for reproducible output. Do not parse help text. +4. Prefer `detect` first on large images, then `recognize --region` on areas of interest. +5. Check exit codes before parsing stdout. Non-zero exit means stdout may be empty; read stderr. diff --git a/.agents/skills/local-ocr/agents/openai.yaml b/.agents/skills/local-ocr/agents/openai.yaml new file mode 100644 index 0000000..2807f85 --- /dev/null +++ b/.agents/skills/local-ocr/agents/openai.yaml @@ -0,0 +1,6 @@ +interface: + display_name: "Local OCR" + short_description: "Offline OCR with coordinates via light-ocr CLI" + default_prompt: "Use $skill-local-ocr to extract text and coordinates from a local image." +policy: + allow_implicit_invocation: true