VLM4OCR v0.6.0

⚡Highlights

OCR Pipelines for heterogeneous documents

OCREngine applies one prompt to every page. That breaks down when a single PDF or TIFF interleaves different page types — for example a packet of cognitive-assessment forms where each form needs its own JSON schema. v0.6.0 adds OCR pipelines: you decide, per page, how that page is processed, while the pipeline owns file loading, concurrency, page ordering, error handling, and OCRResult assembly.

OCREngine.ocr_image_async — the atomic building block

Run a single, already-loaded image through an engine's prompt and get back a standalone OCRPage. No preprocessing is applied (the caller owns rotate/resize), so a page can be OCR'd once and reused across calls.

import asyncio
from PIL import Image
from vlm4ocr import VLLMVLMEngine, OCREngine

engine = OCREngine(VLLMVLMEngine(model="Qwen/Qwen3-VL-30B-A3B-Instruct"), output_mode="markdown")
page = asyncio.run(engine.ocr_image_async(Image.open("page.png")))
print(page.text)

IndependentPagePipeline — per-page processing, concurrently

Supply a process_page(image, *, messages_logger=None) -> OCRPage function and get a full pipeline whose concurrent_ocr mirrors OCREngine.concurrent_ocr (same arguments, same first-complete-first-out AsyncGenerator[OCRResult]). process_page receives only its own page image, so page independence is guaranteed by construction.

import json, asyncio
from vlm4ocr import OCREngine, IndependentPagePipeline, OCRPage

# One classifier + per-type extractors + a fallback — each engine is an atom (one prompt).
classifier = OCREngine(vlm_engine, output_mode="JSON",
                       user_prompt='Classify this page. Respond as JSON: {"page_type": "<one_token>"}')
extractors = {"form_a": OCREngine(vlm_engine, output_mode="JSON", user_prompt="<form A schema>"),
              "form_b": OCREngine(vlm_engine, output_mode="JSON", user_prompt="<form B schema>")}
default = OCREngine(vlm_engine, output_mode="JSON", user_prompt="<generic schema>")

async def classify_and_extract(image, *, messages_logger=None) -> OCRPage:
    label = json.loads((await classifier.ocr_image_async(image, messages_logger=messages_logger)).text)
    page_type = (label[0] if isinstance(label, list) else label).get("page_type", "unknown")
    page = await extractors.get(page_type, default).ocr_image_async(image, messages_logger=messages_logger)
    page.metadata["page_type"] = page_type
    return page

pipeline = IndependentPagePipeline(classify_and_extract, output_mode="JSON")

async def run():
    async for result in pipeline.concurrent_ocr(pdf_paths, concurrent_batch_size=8):
        for page in result.pages:
            print(page.metadata["page_type"], page.text)

asyncio.run(run())

All routing logic — classifier prompt, schemas, type labels — lives in your code; the pipeline only orchestrates. See the OCR Pipelines guide for the full walkthrough and the independence assumption.

OCRPage.metadata

OCRPage now carries a free-form metadata dict (e.g. a page type assigned by a routing pipeline), preserved on each page of the resulting OCRResult. Dict-style access continues to work.

Changes

• New: IndependentPagePipeline for per-page custom processing (routing, classification, multi-pass refinement) with concurrent, files-in / OCRResult-out execution. Exported from the top-level vlm4ocr namespace.
• New: OCREngine.ocr_image_async(image, ...) — the atomic, preprocessing-free single-image OCR call, returning a standalone OCRPage.
• New: OCRPage.metadata free-form dict, and OCRResult.add_page(..., metadata=...).
• New: OCRResult and OCRPage exported from the top-level vlm4ocr namespace.
• Internal: loader dispatch consolidated into vlm4ocr.utils.get_data_loader; SUPPORTED_IMAGE_EXTS now lives in vlm4ocr.utils.
• Compat: fully backward compatible — OCREngine and its existing methods are unchanged.

VLM4OCR v0.5.0

📐Documentation Site

User guide, API reference, and documentation are available at Documentation Page.

⚡Highlights

New bbox output mode

OCREngine now supports output_mode="bbox", which returns the OCR text together with bounding-box coordinates and labels for each detected region. This unlocks two complementary workflows:

• Full-text bbox OCR — leave user_prompt empty (or pass None). The model transcribes the entire page and returns one box per region (line/word/segment, depending on the VLM family).
• Targeted extraction — set user_prompt to a free-text instruction such as "Extract patient name and date of birth". Only regions matching that instruction are returned.

from vlm4ocr import VLLMVLMEngine, OCREngine

vlm_engine = VLLMVLMEngine(model="Qwen/Qwen3-VL-30B-A3B-Instruct")

# Full-text OCR with bounding boxes
ocr = OCREngine(vlm_engine=vlm_engine, output_mode="bbox")
ocr_results = ocr.sequential_ocr(image_path)

# Targeted extraction with bounding boxes
ocr = OCREngine(
    vlm_engine=vlm_engine,
    output_mode="bbox",
    user_prompt="Extract patient name and date of birth",
)
ocr_results = ocr.sequential_ocr(image_path)

Each page in the result exposes a .bboxes list of BBoxItem records (bbox=[x1, y1, x2, y2], label, text) and a .plot_bboxes() helper that returns a PIL.Image.Image with the boxes drawn on top — handy for quickly visualizing or saving annotated pages.

# Inspect bbox results
for page_num, page in enumerate(ocr_results[0].pages):
    for item in page.bboxes:
        print(item.label, item.bbox, item.text)

    # Visualize on the source page
    annotated = page.plot_bboxes()
    annotated.save(f"annotated_page{page_num}.png")

Note that this is a synthesized data with NO real PHI!!

[
	{"bbox_2d": [508, 143, 620, 158], "label": "patient name", "text": "Mary Johnson"},
	{"bbox_2d": [508, 160, 610, 176], "label": "date of birth", "text": "Jan 29, 1990"},
	{"bbox_2d": [484, 405, 569, 421], "label": "Platelets", "text": "280 x10⁹/L"},
	{"bbox_2d": [484, 455, 560, 470], "label": "CMP glucose", "text": "90 mg/dL"},
	{"bbox_2d": [484, 540, 562, 555], "label": "BUN", "text": "12 mg/dL"},
	{"bbox_2d": [484, 778, 690, 794], "label": "RBCs", "text": "0 - 1 per high power field"}
]

Per-VLM bbox formats

Different VLM families emit boxes with different conventions (key names, axis order, coordinate scale). vlm4ocr resolves the right format automatically from the model name through a registry, with built-in support for Qwen3-VL, Gemma 3/4, and GPT-4.1. You can override the resolution by passing a custom BBoxFormat:

from vlm4ocr import OCREngine, BBoxFormat

ocr = OCREngine(
    vlm_engine=vlm_engine,
    output_mode="bbox",
    bbox_format=BBoxFormat(
        coord_scale="normalized_1000",
        axis_order="x0y0x1y1",
        bbox_key="bbox_2d",
        label_key="label",
        text_key="text",
    ),
)

If the model name matches no registered pattern, a default BBoxFormat() is used and a warning is logged.

Web app integration

The Flask web app now exposes BBox as an output format in both the single-file and batch tabs.

• The output area shows an Image | Raw response pill toggle. The Image tab renders bounding boxes on top of the input preview at OCR resolution; the Raw tab streams the model's raw JSON response live during OCR.
• The user-prompt field becomes optional (full-text OCR if empty, targeted extraction if filled).
• Batch mode writes one annotated PNG per page (<stem>_page_<idx>_bbox.png) plus one consolidated <stem>_bbox.json per input file. Both are included in the Download All zip.

OCRPage is now a dataclass

Each entry in OCRResult.pages is now an OCRPage dataclass instead of a plain dict. The new attribute access is the canonical pattern, and dict-style access is preserved for backward compatibility:

page = ocr_results[0].pages[0]

# New attribute-style access (recommended)
page.text
page.image_processing_status
page.bboxes              # populated only in bbox mode
page.image_width         # populated only in bbox mode
page.image_height        # populated only in bbox mode

# Dict-style access still works
page["text"]
page.get("bboxes")

OCRPage also exposes .get_bboxes() and .plot_bboxes() for the bbox workflow.

Changes

• New: output_mode="bbox" on OCREngine for region-level OCR with bounding boxes.
• New: BBoxItem, BBoxFormat, OCRPage exported from the top-level vlm4ocr namespace.
• New: OCRPage.plot_bboxes() returns an annotated PIL.Image.Image for quick visualization.
• New: Web app — BBox output format with image/raw tabbed view, client-side bbox rendering on the input preview, and batch-mode PNG + JSON outputs.
• Compat: OCRResult.pages entries are now OCRPage dataclasses. Dict-style access (page["text"], page.get("bboxes")) continues to work; prefer attribute access (page.text, page.bboxes) for new code.

VLM4OCR v0.4.4

📐Documentation Site

User guide, API reference, and documentation are available at Documentation Page.

⚡Highlights

VLM-based rotation correction

rotate_correction now supports a VLM-based method in addition to Tesseract OSD. This is useful when Tesseract is not installed, or when the input image is too noisy/low-quality for Tesseract to detect orientation reliably.

The rotate_correction parameter accepts "tesseract", "vlm", or False:

from vlm4ocr import VLLMVLMEngine, OCREngine

vlm_engine = VLLMVLMEngine(model="Qwen/Qwen3.5-35B-A3B")
ocr = OCREngine(vlm_engine=vlm_engine, output_mode="markdown")

# Tesseract OSD (default behavior from previous versions)
ocr_results = ocr.sequential_ocr(image_path, rotate_correction="tesseract")

# VLM-based detection — reuses the OCR engine's VLM
ocr_results = ocr.sequential_ocr(image_path, rotate_correction="vlm")

FewShotExample accepts the same options. When rotate_correction="vlm", pass the vlm_engine so the example image can be corrected at construction time:

from PIL import Image
from vlm4ocr import FewShotExample, VLLMVLMEngine

vlm_engine = VLLMVLMEngine(model="Qwen/Qwen3.5-35B-A3B")

example = FewShotExample(
    image=Image.open("example_1.JPG"),
    text="<Expected OCR output for example 1>",
    rotate_correction="vlm",
    vlm_engine=vlm_engine,
    max_dimension_pixels=512,
)

The CLI exposes the same choice:

vlm4ocr --rotate_correction vlm ...

Changes

• Breaking: rotate_correction no longer accepts True/False booleans. Use "tesseract" for the previous default behavior, "vlm" for the new method, or False to disable.
• ImageProcessor has been moved from vlm4ocr.utils into a new vlm4ocr.preprocessing module. Update imports to from vlm4ocr.preprocessing import ImageProcessor.

VLM4OCR v0.4.3

📐Documentation Site

User guide, API reference, and documentation are available at Documentation Page.

Changes

Added support for SGLang

from vlm4ocr import SGLangVLMEngine

vlm_engine = SGLangVLMEngine(model="Qwen/Qwen3-VL-30B-A3B-Instruct")

OCR engine now allows no system prompt/ user prompt

For some OCR models (e.g., PaddleOCR, LightOn-OCR), system prompt and user prompt are not required. The OCR engine now allows both to be empty.
Setting system_prompt and user_prompt to False will avoid using them. Note that Leaving system_prompt and user_prompt None will use default prompts.

from vlm4ocr import OCREngine

ocr_engine = OCREngine(vlm_engine=vlm_engine, system_prompt=False, user_prompt=False)

Added graceful shutdown for VLM engines

concurrent_ocr() now cancels all in-flight VLM calls when the consumer stops iterating — via break, an exception, or aclose(). Previously, abandoning the generator left background tasks running until they completed, wasting API calls and compute.

Python code — breaking out of the loop now cleanly stops remaining files:

async for result in ocr_engine.concurrent_ocr(file_paths):
    if some_condition:
        break  # remaining VLM calls are cancelled immediately
    process(result)

CLI — Ctrl+C now exits cleanly without leaving orphaned VLM calls in flight.

Web app — the Stop button in the UI aborts both single-file and batch OCR mid-run.

VLM4OCR v0.4.2

📐Documentation Site

User guide, API reference, and documentation are available at Documentation Page.

Changes

Changed internal interface of inference engines

• The internal interface of all inference engines has been updated to be compliant with new version of dependency llm-inference-engine v0.1.5

VLM4OCR v0.4.1

📐Documentation Site

User guide, API reference, and documentation are available at Documentation Page.

Changes

• Adopted llm-inference-engine to implement VLMEngine for better modularity and maintainability.
• All interfaces and functionalities remain unchanged.

VLM4OCR v0.4.0

📐Documentation Site

User guide, API reference, and documentation are available at Documentation Page.

⚡Highlights

Added few-shot examples support

Supply few-shot examples to improve OCR accuracy:

from PIL import Image
import asyncio
from vlm4ocr import FewShotExample, AzureOpenAIVLMEngine, OCREngine

# Load few-shot examples
# Note that the example text should be the expected OCR output for the example image. Do not include any extra instructions.
example_1_image = Image.open("example_1.JPG"))
example_1_text = """<Expected OCR output for example 1>"""

example_2_image = Image.open("example_2.JPG")
example_2_text = """<Expected OCR output for example 2>"""

few_shot_examples = [
    FewShotExample(image=example_1_image, text=example_1_text, max_dimension_pixels=512),
    FewShotExample(image=example_2_image, text=example_2_text, max_dimension_pixels=512)
]

We load the target image for OCR.

image_path = os.path.join("examples", "synthesized_data", "few_shot_examples", "images", "template_6_sample_4_poor.JPG")

Define the VLM engine and OCR engine.

from vlm4ocr import VLLMVLMEngine, OCREngine

# Define VLM engine
vlm_engine = VLLMVLMEngine(model="Qwen/Qwen2.5-VL-7B-Instruct")

# Define OCR engine
ocr = OCREngine(vlm_engine, output_mode="text")      

Now, we pass the few-shot examples to the OCR methods.

# OCR for a single image
ocr_results = ocr.sequential_ocr(image_path, max_dimension_pixels=512, verbose=True, few_shot_examples=few_shot_examples)

Added concurrent OCR pipeline template and config

To run OCR on multiple images concurrently, define a YAML config file and run the pipeline. This serves as a template for building custom OCR pipelines.

---
  run_name: "default_run"
  # VLM Engine Configuration
  vlm_engine:
    model: "<your-model-name>"
    base_url: "<your-vlm-endpoint-url>"
    config: BasicVLMConfig(max_new_tokens=8192)
  # OCR Engine Configuration
  ocr_engine:
    # OCR output format
    output_mode: "markdown"
    # number of pages to process in concurrent 
    concurrent_batch_size: 8
    # maximum number of files to load into memory at once
    max_file_load: 16
    # user prompt file path
    user_prompt_path: <path-to-your-prompt-file>

  input_directory: "<path-to-your-input-directory>"
  output_directory: "<path-to-your-output-directory>"
  messages_log_directory: "<path-to-your-messages-log-directory>"

VLM4OCR v0.3.1

📐Documentation Site

User guide, API reference, and documentation are available at Documentation Page.

⚡Highlights

Added reasoning VLM supports

Given the new reasoing VLMs (e.g., Qwen-VL-30B-A3B-Thinking, o4-mini), we added new configs to support these models.

from vlm4ocr import OpenAIVLMEngine, VLLMVLMEngine, OCREngine, OpenAIReasoningVLMConfig, ReasoningVLMConfig

# vLLM
config = ReasoningVLMConfig(temperature=0.6, top_p=0.95)
engine = VLLMVLMEngine(model="Qwen/Qwen3-VL-30B-A3B-Thinking", config=config)
# OpenAI
config = OpenAIReasoningVLMConfig(reasoning_effort="low")
engine = OpenAIVLMEngine(model="o4-mini", config=config)

ocr = OCREngine(vlm_engine=engine, output_mode="markdown")
ocr_results = ocr.sequential_ocr([image_path_1, image_path_2], verbose=True)

Added OpenAICompatible VLM engines

We separated the OpenAI compatible VLM engines into a parent class OpenAICompatibleVLMEngine to allow for easier integration with other inferencing services that follow the OpenAI chat-completion API format. For vLLM, use VLLMVLMEngine. For OpenRouter, use OpenRouterVLMEngine. For other services that follow the OpenAI chat-completion API format, you can create your own engine by inheriting from OpenAICompatibleVLMEngine.

from vlm4ocr import VLLMVLMEngine, OpenRouterVLMEngine, OCREngine

# VLLM
engine = VLLMVLMEngine(model="Qwen/Qwen3-VL-30B-A3B-Instruct")
# OpenRouter
engine = OpenRouterVLMEngine(model="Qwen/Qwen3-VL-30B-A3B-Instruct", api_key="YOUR_API_KEY")

ocr = OCREngine(vlm_engine=engine, output_mode="markdown")
ocr_results = ocr.sequential_ocr([image_path_1, image_path_2], verbose=True)

Changes

• Added message logging to OCR results for better debugging and tracking of the OCR cost.

import json
from vlm4ocr import VLLMVLMEngine, OCREngine

vllm = VLLMVLMEngine(model="Qwen/Qwen3-VL-30B-A3B-Instruct")
ocr = OCREngine(vlm_engine=vllm, output_mode="markdown")
ocr_results, messages_log = ocr.sequential_ocr([image_path_1, image_path_2], verbose=True)

print(json.dumps(ocr_results[0].get_messages_log(), indent=4))
print(json.dumps(ocr_results[1].get_messages_log(), indent=4))

VLM4OCR v0.3.0

📐Documentation Site

User guide, API reference, and documentation are available at Documentation Page.

⚡Highlights

Added key information extraction with JSON

In some use cases, we are only interested in a specific set of key information from the OCR results. Processing the entire OCR text is inefficient. We can directly extract the key information using the output_mode="JSON". To use the JSON extraction feature, a custom user prompt that defines the JSON structure is required. The example below demonstrates how to extract key information from images and PDFs.

Run OCR sequentially

import json
from vlm4ocr import OCREngine

user_prompt = """
Your output should include keys: "Patient", "MRN". 
For example:
{
    "Patient": "John Doe",
    "MRN": "12345"
}
"""

ocr = OCREngine(vlm_engine=vlm, output_mode="JSON", user_prompt=user_prompt)
ocr_results = ocr.sequential_ocr([image_path_1, image_path_2], verbose=True)

for result in ocr_results:
    for page_num, page in enumerate(result.pages):
        print(json.loads(page['text']))
        with open(f"{result.filename}_page_{page_num}.json", "w", encoding="utf-8") as f:
            json.dump(json.loads(page['text']), f, indent=4)

Run OCR concurrently

import asyncio
import json
from vlm4ocr import OCREngine

user_prompt = """
Your output should include keys: "Patient", "MRN". 
For example:
{
    "Patient": "John Doe",
    "MRN": "12345"
}
"""
ocr = OCREngine(vlm_engine=vlm, output_mode="JSON", user_prompt=user_prompt)

async def run_ocr():
    response = ocr.concurrent_ocr([image_path_1, image_path_2], concurrent_batch_size=4)
    async for result in response:
        if result.status == "success":
            filename = result.filename
            for page_num, page in enumerate(result.pages):
                with open(f"{filename}_page_{page_num}.json", "w", encoding="utf-8") as f:
                    json.dump(json.loads(page['text']), f, indent=4)
                print(f"Saved {filename}_page_{page_num}.json")
        else:
            print(f"Error processing {result.filename}: {result.error}")

asyncio.run(run_ocr())

Added batch processing to web app

The web app now supports concurrent batch processing of multiple images using OCREngine.concurrent_ocr method as backend.

Changes

• Added missing dependency colorama

VLM4OCR v0.2.0

📐Documentation Site

User guide, API reference, and documentation are available at Documentation Page.

⚡Highlights

Added image processing features

Added rotate_correction that use Tesseract to correct for rotation. max_dimension_pixels resize images to ensure the largest dimension (width or length) are less than the maximum allowed pixels. This manages the input image tokens.

Image processing with Python

response = ocr.concurrent_ocr(file_paths=<a list of files>, 
                              rotate_correction=True,
                              max_dimension_pixels=4000,
                              concurrent_batch_size=4,
                              max_file_load=8)

Image processing in CLI

vlm4ocr --input_path /examples/synthesized_data/ \
        --output_mode markdown \
        --max_dimension_pixels 4000 \
        --rotate_correction \
        --vlm_engine openai_compatible \
        --model Qwen/Qwen2.5-VL-7B-Instruct \
        --api_key EMPTY \
        --base_url http://localhost:8000/v1 \
        --concurrent_batch_size 4

Implemented asynchronous OCR output

We added concurrent_ocr method to OCREngine that returns an async generator of OCRResult instance (AsyncGenerator[OCRResult, None]). OCR results are generated whenever is ready (first-done-first-out). There is no guarantee the input order and output order will match. Use the OCRResult.filename as identifier.

Example: dynamic output-writing

The example below use concurrent_ocr to perform OCR and write available results to file.

import asyncio

async def run_ocr():
    response = ocr.concurrent_ocr(<list of files>, concurrent_batch_size=4)
    async for result in response:
        if result.status == "success":
            filename = result.filename
            ocr_text = result.to_string()
            with open(f"{filename}.md", "w", encoding="utf-8") as f:
                f.write(ocr_text)

asyncio.run(run_ocr())

Optimized file staging

We added max_file_load parameter to concurrent_ocr method. The max_file_load manages files pre-loading (staged and waiting for OCR). The concurrent_batch_size manages the number of images/pages VLM processes at a time. The max_file_load and concurrent_batch_size should be tuned to optimize system performance and avoid bottleneck.

Changes

Moved VLM sampling parameters to VLMConfig

To support different VLMs, especially reasoning models, we moved sampling parameters (e.g., temperature, max_new_tokens) from OCREngine to VLMConfig. Now, those parameters can be set by passing a configuration instance to the VLMEngine.

from vlm4ocr import BasicVLMConfig, OpenAIVLMEngine

vlm_engine = OpenAIVLMEngine(model="Qwen/Qwen2.5-VL-7B-Instruct", 
                             base_url="http://localhost:8000/v1", 
                             api_key="EMPTY",
                             config=BasicVLMConfig(max_tokens=4096, temperature=0.0)    
                            )

For reasoning models (e.g., o3-mini), use OpenAIReasoningVLMConfig to set reasoning effort.

from vlm4ocr import OpenAIReasoningVLMConfig, OpenAIVLMEngine

vlm_engine = OpenAIVLMEngine(model="o3-mini", config=OpenAIReasoningVLMConfig(reasoning_effort="low") )