Two questions, answered with numbers anyone can regenerate:
- Accuracy — if the same document enters a RAG pipeline as a PDF or as a JDF, which one lets the retriever find the right passage? (
rag_bench.py) - Cost — what do 1,000 PDF files vs 1,000 JDF files cost to index, re-index and query through the same RAG pipeline? (
cost_bench.py)
Everything else is held equal: same content, same embedding models, same lexical retriever, same questions, same hit rule. Latest numbers: results/report.md · machine-readable with per-question ranks: results/latest.json, results/cost-latest.json · web: uurtech.github.io/jdf/docs/benchmark.
git clone https://github.com/uurtech/jdf && cd jdf/bench
pip install -r requirements.txt
# fresh (GPU) machine, one shot — venv, CUDA torch, deps, GPU check, both benchmarks:
./run_gpu.sh # add --fresh to recompute every embedding; EMBEDDERS=… FILES=… to override
python rag_bench.py # BM25 + local embeddings (default: BAAI/bge-small-en-v1.5)
python rag_bench.py --embedder ollama:nomic-embed-text,st:BAAI/bge-base-en-v1.5
python rag_bench.py --embedder none # BM25 only — no model download, runs in seconds
python cost_bench.py # RAG cost, 1,000 files per format (--files 10000 for more)
python rag_bench.py --verify # re-run and compare with results/latest.json
# GPU: the sentence-transformers embedder picks CUDA automatically and prints
# "embedder st:<model>: cuda:0 (<GPU name>)" at start; force with RAG_BENCH_DEVICE=cuda:1|cpu.
# Cached vectors in .cache/emb-*.json are reused and never touch the GPU — delete them for a real GPU run.
python cost_bench.py --verifyNo Node toolchain needed: the JDF side is the committed output of jdf chunk (corpus/jdf-chunks.jsonl). Each installed PDF parser becomes a pipeline; missing ones are skipped and reported:
| Input | Install |
|---|---|
| PyMuPDF, pdfplumber, pypdf | in requirements.txt |
poppler pdftotext |
brew install poppler / apt install poppler-utils (optional) |
| sentence-transformers models | downloaded from Hugging Face on first use; --embedder none to skip |
| Ollama models | ollama pull nomic-embed-text, then --embedder ollama:nomic-embed-text |
Nothing from the corpus leaves the machine. The cost benchmark never calls a paid API — dollars are counted tokens × the prices in prices.json (each with an as-of date and source; edit for your provider).
Corpus. 24 "Quarterly Operations Report" documents, 4–5 pages each, generated deterministically from seed 20260912 by src/gen-corpus.ts. Each has eight sections: two prose sections, five tables (regional performance, headcount, pricing, service levels, vendor spend) and a risk list, plus a repeating page header and footer. Company names, numbers, people and dates are synthetic. The JDF files are the originals; the PDFs in corpus/docs/ were printed from them by the real jdf.js renderer in Google Chrome (src/print-pdf.mjs), so the PDF text layer is clean and complete — the PDF side is not handicapped by scans or exotic fonts.
Questions. 192, generated with the corpus and stored with ground truth in corpus/questions.json: 120 table-cell lookups ("What was Acme Logistics's churn rate in EMEA for Q3 2025?"), 48 prose facts (on-call lead, capex budget), 24 list items (risk owner). Every question names the document the way a user would.
PDF pipelines. pdf → text → fixed-size chunks → retriever. Parsers: PyMuPDF, pdfplumber, pypdf, poppler pdftotext -layout. Chunker: a faithful implementation of LangChain's RecursiveCharacterTextSplitter (separators \n\n, \n, space, character) at the two most common settings, 1000/200 and 2000/200 characters. The landing page shows each parser's best.
JDF pipeline. jdf chunk --strategy section --max-tokens 512, i.e. exactly what the CLI ships (corpus/jdf-chunks.jsonl, regenerated by pnpm --filter @jdf/bench export-chunks). Section chunks follow the heading hierarchy, tables are serialised as Header: value | Header: value rows so a number never loses its column, and the heading breadcrumb is prefixed to the text that gets embedded/indexed — the same string jdf embed sends to the model.
Converted pipeline. pdf → jdf convert → jdf chunk: the same PDFs run through the shipped importer (packages/jdf-pdf-import, which rebuilds tables into real table elements and detects headings from font size), then chunked exactly like native JDF (corpus/jdf-chunks.converted.jsonl, regenerated by export-chunks). This is the realistic path for a PDF archive — and it scores like native JDF.
Retrievers. Applied identically to every pipeline: BM25 (Okapi, k1 = 1.5, b = 0.75, zero dependencies, deterministic) and cosine search over embeddings from any sentence-transformers or Ollama model (--embedder takes a comma-separated list; model-specific query/document prefixes are applied to both sides).
Hit rule. A retrieved chunk is a hit when it comes from the right document and its text contains both the answer string and the row/subject key (whitespace-insensitive).
Metrics. Recall@1/3/5, MRR@10, and R@1k tokens / R@2k tokens — was the answer inside the first 1,000 / 2,000 tokens of retrieved context. The token-budget metric is chunk-size neutral: a 2,000-character PDF chunk is a quarter of the document, so it "hits" more often at rank 1 and then costs three times the tokens per query. Also reported per question type, plus the tokens an LLM would receive for the top-5 chunks.
JDF-only properties. Every committed chunk's hash is checked against sha256(text); one paragraph of one document is edited and re-chunked (corpus/jdf-chunks.d01-edited.jsonl) — the number of chunks whose hash changed is what jdf embed --incremental re-embeds. A PDF pipeline has no stable chunk identity, so any edit means re-extract, re-chunk, re-embed.
RAG cost (cost_bench.py). The corpus is cycled to N files per format (default 1,000; the published run uses --files 10000). Both sides run the same pipeline — chunks → embeddings → vector store → top-5 context → LLM — and only the input format differs; the PDF column is the PDF pipeline that scored best in the accuracy run. Counted: chunks and embedding tokens for the initial index (ceil(chars/4), both sides), vector-store payload, re-index tokens when one paragraph changes in every document (JDF: only hash-changed chunks; PDF: the whole document), LLM input tokens for the query volume (--queries, default 1M; the published run uses 10M) with top-5 context. Measured: local embedding throughput on a real sample of each pipeline's chunks. Dollar rows use prices.json. Accuracy from the same run is repeated in the table so cost and quality sit side by side.
The generated corpus gives exact ground truth; byoc.py answers "and on real PDFs?" with the same pipelines, retrievers and hit rule on public documents (byoc/corpus.json: arXiv papers including a two-column one, two AWS whitepapers — downloaded at run time, never committed) or on any folder you pass with --pdf. Questions are derived from the documents deterministically: a sentence that states a number is turned into a fill-in-the-blank, and a hit must contain the number and the sentence's key word in the right document. The JDF side is jdf convert + jdf chunk run on the PDF (src/byoc-chunks.ts). Unsupervised questions are noisier than labelled ones, so absolute numbers are lower and both sides share that noise.
| Pipeline | Chunks | BM25 R@1k tok | nomic-embed-text R@1k tok | nomic-embed-text top-1 | Ctx tokens @5 |
|---|---|---|---|---|---|
| PDF → jdf convert → jdf chunk (section, 512 tok) | 570 | 94.5% | 64.8% | 54.7% | 2047 |
| PDF → jdf convert → jdf chunk (section, 256 tok) | 871 | 96.9% | 80.5% | 57.0% | 1075 |
| PDF · PyMuPDF get_text() · fixed 1000/200 | 856 | 97.7% | 78.9% | 57.8% | 1210 |
| PDF · PyMuPDF get_text() · fixed 2000/200 | 393 | 97.7% | 64.1% | 50.0% | 2373 |
| PDF · pdfplumber extract_text() · fixed 1000/200 | 843 | 82.0% | 77.3% | 57.0% | 1196 |
| PDF · pdfplumber extract_text() · fixed 2000/200 | 386 | 80.5% | 63.3% | 43.0% | 2356 |
| PDF · pypdf extract_text() · fixed 1000/200 | 854 | 97.7% | 81.2% | 56.2% | 1207 |
| PDF · pypdf extract_text() · fixed 2000/200 | 391 | 96.9% | 60.2% | 44.5% | 2389 |
| PDF · pdftotext -layout (poppler) · fixed 1000/200 | 1098 | 96.1% | 80.5% | 61.7% | 1119 |
| PDF · pdftotext -layout (poppler) · fixed 2000/200 | 460 | 98.4% | 67.2% | 56.2% | 2267 |
Real public PDFs — BERT: Pre-training of Deep Bidirectional Transformers, Attention Is All You Need, Overview of Amazon Web Services, Serverless Applications Lens (297 pages) — with 128 numeric-fact questions derived from the documents themselves (see bench/byoc.py; unsupervised, so absolute numbers are lower than the labelled benchmark and both sides share the noise). JDF side = jdf convert + jdf chunk 0.2.2, the same code the CLI and reader ship. Apple M5, 2026-09-18. Re-run: python bench/byoc.py --embedder ollama:nomic-embed-text; your own folder: --pdf DIR.
Results depend on the embedding model. With nomic-embed-text, bge-base and BM25, JDF leads on every metric. With the two smallest models (bge-small, all-MiniLM-L6-v2) the PDF 2,000-character configuration edges out JDF at top-1 — because each of its chunks is a quarter of the document — while JDF still leads on R@1k tokens with every retriever and hands the LLM roughly a third of the tokens. Both views are in results/report.md; the landing page defaults to the chunk-size-neutral metric and lets you switch.
A retrieval benchmark needs to know, for every question, which passage answers it. Public PDF collections do not ship that, and hand-labelling hundreds of answers is where benchmark bias usually creeps in. Generating the documents makes the ground truth exact and the whole thing auditable: read the generator, open any document, open any question, check any rank in results/latest.json. --verify proves nobody edited a document, a question or a chunk after the fact by comparing SHA-256 hashes with corpus/manifest.json.
The obvious limitation: synthetic reports are cleaner than real ones. Real PDFs are usually worse for the PDF side (scans, multi-column layouts, broken tables), so treat these numbers as a floor for the gap, not a ceiling. Running the same pipelines on your own corpus is the next step — open an issue with what you find.
bench/
rag_bench.py accuracy benchmark (Python, standalone)
cost_bench.py RAG cost benchmark at 1,000 files (Python, standalone)
prices.json public list prices used for $ rows (as-of + source)
requirements.txt
corpus/docs/dNN.jdf, dNN.pdf generated documents (committed)
corpus/questions.json 192 questions with ground truth
corpus/jdf-chunks.jsonl `jdf chunk` output for every document (+ d01-edited variant)
corpus/jdf-chunks.converted.jsonl same, for the PDFs converted back with `jdf convert`
corpus/manifest.json seed + SHA-256 of every file
results/latest.json accuracy results incl. per-question ranks
results/cost-latest.json cost results
results/report.md human-readable tables
byoc.py, byoc/corpus.json real-corpus benchmark (public PDFs downloaded to byoc/cache/, gitignored)
results/byoc-latest.json real-corpus results
src/ maintainer tooling (Node): gen-corpus.ts, print-pdf.mjs, export-chunks.ts, byoc-chunks.ts, render.mjs
Maintainers: after changing the generator or the CLI chunker — pnpm --filter @jdf/bench corpus && pnpm --filter @jdf/bench print && pnpm --filter @jdf/bench export-chunks, then re-run both Python benchmarks and pnpm --filter @jdf/bench render to publish the numbers to README, landing page and docs.