Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
156 changes: 88 additions & 68 deletions .agents/skills/local-ocr/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,106 +1,126 @@
# light-ocr: Local OCR Skill
---
name: local-ocr
description: Extract text from local images (PNG, JPEG) with precise coordinates, confidence scores, and stable error handling using the light-ocr CLI. Use when needing to read small/dense text in screenshots, receipts, labels, forms, or documents that a multimodal model may misread; when exact text plus bounding box coordinates are needed for field extraction, redaction, counting, or layout analysis; or when deterministic offline OCR is required without network or API calls.
---

Use this skill when you need to extract text from local images (PNG, JPEG) with precise coordinates, confidence scores, and stable error handling — without writing Node.js integration code or relying on a multimodal model's ability to read small text.
# local-ocr

## When to use OCR instead of a multimodal model
`light-ocr` is a local OCR engine. It runs offline, returns text with coordinates and confidence, and follows a strict stdout/stderr contract for scripting.

- Small text, dense text, or text in screenshots/labels/receipts/forms that a multimodal model misreads or hallucinates.
- When you need exact text + bounding box coordinates (for field extraction, redaction, counting, or downstream layout analysis).
- When you need deterministic, offline, reproducible results (no network, no API calls).
## Scenarios

## Commands
### Screenshot with small text

A user shares a screenshot and asks about specific text that is too small or dense to read visually.

```bash
# Full OCR: recognize text + coordinates (default action)
light-ocr image.png --format json
light-ocr image.png --format text # just text, no coordinates
# Step 1: recognize the full image
light-ocr screenshot.png --format json
```

If the result has low confidence or missing text in a region:

# Region-only recognition (ROI)
light-ocr recognize image.png --region 100,80,640,320 --format json
```bash
# Step 2: re-run on the specific region (coordinates from step 1 boxes)
light-ocr recognize screenshot.png --region 100,80,640,320 --format json
```

# Detect-only: just text region boxes, no recognition
light-ocr detect image.png # output is always JSON
light-ocr detect image.png --crop # include PNG crop per box
### Form or receipt field extraction

# Diagnostics (no image read)
light-ocr info --model-info # full EngineInfo JSON
light-ocr info --version # npm/core/model triple
Need to extract specific fields (names, amounts, dates) from a form or receipt image.

# stdin
cat image.png | light-ocr recognize --stdin --type image/png --format json
```bash
# Step 1: detect where text regions are (fast, no recognition)
light-ocr detect receipt.png

# Execution provider
light-ocr recognize image.png --provider auto # default: auto-select best
light-ocr recognize image.png --provider cpu # force CPU
# Step 2: recognize only the region containing the target field
light-ocr recognize receipt.png --region 50,200,300,80 --format json
```

## Output schema
This two-step pattern saves time on large images: detect first, then recognize only the regions of interest.

### Counting text regions

Need to count how many text lines or regions exist in an image.

```bash
light-ocr detect image.png | python -c "import json,sys; print(len(json.load(sys.stdin)['pages'][0]['detections']))"
```

### Verifying multimodal model output

A multimodal model claims to read text from an image. Verify the claim against deterministic OCR.

```bash
light-ocr image.png --format text
```

Compare the text output with the model's claim. If they differ, trust the OCR `text` field — do not fabricate.

All `recognize`/`detect` output uses `--schema-version 1` (default). The envelope:
### Batch processing via shell

Process multiple images sequentially with JSONL output.

```bash
for f in *.png; do
light-ocr recognize "$f" --format jsonl
done
```

Each line is one page record. Check exit codes: a non-zero exit for one image does not stop the loop, but stdout for that image may be empty.

## Decision flow

```
Need text from an image?
├── Know which region? → recognize --region x,y,w,h --format json
├── Need full text only? → recognize --format text
├── Need text + coordinates? → recognize --format json
├── Only need where text is? → detect
├── Large image, unsure where text is? → detect first, then recognize --region
└── Need engine info or version? → info --model-info / info --version
```

## Output schema

```json
{
"schemaVersion": 1,
"source": { "kind": "image", "mediaType": "...", "identity": {}, "appliedTransforms": {} },
"pages": [{ "index": 0, "width": ..., "height": ..., "coordinateSpace": "pageSpace", "structure": "ocr-order|detect", "lines|detections": [] }]
"pages": [{
"index": 0,
"width": 640, "height": 480,
"coordinateSpace": "pageSpace",
"structure": "ocr-order",
"lines": [{ "id": "L0", "text": "HELLO", "confidence": 0.99, "box": [4 points] }]
}]
}
```

- `recognize`: `pages[0].lines[]` with `{ id: "L0", text, confidence, box: [4 points] }`
- `detect`: `pages[0].detections[]` with `{ id: "D0", score, box: [4 points] }`
- `--format text`: only recognized text, one line per line (no coordinates)
- `box` is 4 points in `pageSpace` (top-left origin, x right, y down, post-EXIF pixels)
- `detect` replaces `lines` with `detections[]` (`{ id, score, box }`) and sets `structure: "detect"`
- `--format text`: recognized text only, one line per line, no coordinates
- `--format jsonl`: one page record per line (for streaming/batch)

## Choosing what to run

| You want | Command |
| --- | --- |
| Full text from an image | `recognize --format text` |
| Text + coordinates | `recognize --format json` |
| Just where text is (no text) | `detect` |
| Text in a specific area | `recognize --region x,y,w,h` |
| Engine/provider info | `info --model-info` |
| Quick version check | `info --version` |

## Exit codes

| Code | Meaning | Agent action |
| Code | Meaning | Action |
| --- | --- | --- |
| 0 | Success | Parse stdout |
| 64 | Usage error | Fix command syntax |
| 65 | Invalid argument (bad region, unsupported format/schema) | Fix input |
| 66 | Invalid image | Try different image |
| 67 | Unsupported capability | Check `info --model-info` |
| 68 | Model/bundle error | Reinstall package |
| 69 | Resource limit exceeded | Smaller image or region |
| 69 | Resource limit exceeded | Use smaller image or `--region` |
| 70 | Environment/package failure | Check native addon |
| 71 | Inference failure | Retry or report bug |
| 72 | Internal error | Report bug |

## How to handle failures

- **Empty result**: No text found. The image may have no text, or text is too small/low-contrast. Try `--region` on specific areas, or check with `detect` first.
- **Low confidence**: Lines with `confidence < 0.5` may be unreliable. Don't present inferred or guessed text as OCR output — always cite the actual `text` field and its `confidence`.
- **Resource limit (exit 69)**: Image too large. Use `--region` to process a sub-area, or downscale before passing.
- **Unsupported capability (exit 67)**: The requested provider or feature isn't available. Run `info --model-info` to see what's supported.

## Important rules

1. **Never fabricate OCR text.** Only use text from the `text` field of the result. If confidence is low, say so — don't guess.
2. **Cite coordinates when relevant.** Box coordinates are in `pageSpace` (top-left origin, x right, y down, post-EXIF pixels). Use them for field extraction, redaction, or counting.
3. **Use `--schema-version 1`** for reproducible output. Don't parse help text programmatically.
4. **Prefer `detect` first** if you only need to locate text regions (faster, no recognition).
5. **Use `--region`** to avoid processing huge images unnecessarily — detect first, then recognize specific regions.
6. **Check exit codes** before parsing stdout. Non-zero exit means stdout may be empty; stderr has the error.

## Validation script

```bash
# Quick smoke test: recognize a known image and check exit code
light-ocr test-image.png --format text && echo "OK: $(light-ocr test-image.png --format text | wc -l) lines"
```

## Related
## Rules

- [CLI design](docs/cli-design.md) — full flag reference, coordinate semantics, exit codes
- [Roadmap N1](docs/roadmap.md) — product context and acceptance criteria
1. Never fabricate OCR text. Only use the `text` field from results. If confidence < 0.5, state it.
2. Cite coordinates when relevant. Box coordinates are in `pageSpace`.
3. Use `--schema-version 1` for reproducible output. Do not parse help text.
4. Prefer `detect` first on large images, then `recognize --region` on areas of interest.
5. Check exit codes before parsing stdout. Non-zero exit means stdout may be empty; read stderr.
6 changes: 6 additions & 0 deletions .agents/skills/local-ocr/agents/openai.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
interface:
display_name: "Local OCR"
short_description: "Offline OCR with coordinates via light-ocr CLI"
default_prompt: "Use $skill-local-ocr to extract text and coordinates from a local image."
policy:
allow_implicit_invocation: true
Loading