LiteLLM IDP test

Run the IDP (intelligent document processing) test that transcribes a PDF via Gemini and checks it against expected text.

Setup

python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -r requirements.txt
export GEMINI_API_KEY=your-key-here

Run

python idp-gemini-test.py

Optional Arguments

All flags below override the script defaults.

--pdf : Input PDF path (default: ./test-1.pdf).

--expected : Expected text file to compare against (default: ./test-1.txt).

--model : LiteLLM model name (default: IDP_MODEL env var if set, otherwise gemini/gemini-2.5-flash).

--temperature : Sampling temperature as a float (default: 0.0).

--out-json : Where to write the validated transcription JSON (default: ./idp-<test-name>-response.json next to the script, derived from --expected).

--fill-missing-notes : If the model omits the optional notes field, set it to a deterministic value so the saved JSON always includes notes (useful for exercising the field).

The prompt asks the model to include optional notes (e.g. uncertainty or observations). If the response omits or leaves notes empty, the script prints a warning to stderr but does not fail; use --fill-missing-notes to auto-fill and ensure the output JSON has a notes key.

Text comparison is done after normalization (trimming and collapsing whitespace).

Exit code is 0 on match, 1 on mismatch or API/schema errors, and 2 on missing GEMINI_API_KEY or missing input files.

Example:

python idp-gemini-test.py --pdf ./test-1.pdf --expected ./test-1.txt --model gemini/gemini-2.5-flash --temperature 0.0 --out-json ./out.json