ocr-svc

A small OCR microservice. You POST an image, you get its text back — one page per call.

It runs PP-OCRv6 on ONNX Runtime, with a hybrid CPU + GPU setup so pages can be processed on whichever lane is free. It's built for a caller that fans out the pages of a PDF and hits the API once per page, concurrently.

Every response is the same three keys:

{ "text": "…", "error": null, "time_taken": "MM:SS.cc" }

• Success → 200, error: null.
• No text found → 200, error: "NO_TEXT_FOUND".
• Failure → a real HTTP status + an error code, e.g. IMAGE_TOO_LARGE (413), INVALID_IMAGE (422), TIMEOUT (504), SERVICE_UNAVAILABLE (503).

Nothing ever throws out of the pipeline — a bad page fails on its own, the rest keep going.

Architecture

   ┌───────────────────────────────────────────────────────────┐
   │  Client  (a PDF pipeline fanning out pages, one call each) │
   └─────────────────────────────┬─────────────────────────────┘
                                 │  POST /v1/ocr  (multipart image)
                                 ▼
   ┌───────────────────────────────────────────────────────────┐
   │  API — FastAPI / uvicorn                                   │
   │  validate → sha256 dedup → enqueue → await result          │
   └───────┬───────────────────────────────────┬───────────────┘
           │ dedup + single-flight             │ enqueue / await
           ▼                                   ▼
   ┌───────────────┐                   ┌───────────────┐
   │   Postgres    │                   │     Redis     │
   │ dedup + logs  │                   │   job queue   │
   └───────────────┘                   └───┬───────┬───┘
                                           │       │   Redis work-steals:
                              ┌────────────▼─┐   ┌─▼────────────┐  whichever
                              │  worker-gpu  │   │  worker-cpu  │  lane is free
                              │ resident     │   │ process pool │  pulls the job
                              │ ONNX model   │   │ ONNX model   │
                              └───────┬──────┘   └──────┬───────┘
                                      └──── PP-OCRv6 ────┘
                                     det → rec (+ orient / unwarp)
                                              │
                                results → .txt volume + Postgres record

   Observability:  all services → OpenTelemetry Collector
                     → Prometheus (metrics) · Loki (logs) · Tempo (traces) → Grafana

The API is thin: it validates the image, deduplicates by content hash, drops the job on a Redis queue, and waits for the answer. Two kinds of workers listen on that same queue — a GPU worker with the model resident in VRAM, and a CPU worker running a process pool — and Redis hands each job to whichever is free. Identical pages in flight collapse into a single inference (single-flight), and repeat pages are served straight from the cache.

Quick start

Needs Docker. For the GPU lane you also need the NVIDIA Container Toolkit.

# full stack + GPU lane
docker compose --profile gpu up -d

# or CPU only
docker compose up -d

Then:

• API — http://localhost:8000/docs
• Grafana — http://localhost:3000 (admin / admin)
• Prometheus — http://localhost:9090

Check it's healthy with curl http://localhost:8000/healthz.

Usage

python example.py page.png

Or with curl:

curl -s -F file=@page.png http://localhost:8000/v1/ocr
# {"text":"…","error":null,"time_taken":"00:01.92"}

example.py prints the full text; tests/test_client.py fires many concurrent requests to simulate a PDF fan-out (CONCURRENCY=100 python tests/test_client.py page.png --repeat 200).

Endpoints: POST /v1/ocr, and GET /healthz, /readyz, /metrics.

Config & observability

Settings live in config.yaml — the model tier (tiny / small / medium), the CPU pool size, the GPU lane, the request timeout. It's baked into the images at build time, so change it and rebuild (docker compose --profile gpu build) to pick it up.

Grafana ships with a preloaded ocr-svc dashboard: requests/sec per lane, cache-hit ratio, latency p50/p95/p99, queue depth, errors by type, GPU utilisation and VRAM, host CPU/RAM, and a live log panel. It's the fastest way to see what the service is doing.

Docs

More detail lives in docs/:

• Architecture — request lifecycle, lanes, dedup & single-flight, failure model.
• Operations — deploy, GPU setup, config reference, changing the model, troubleshooting.
• API reference — endpoints, error codes, limits, examples.