VeriQuote

Local quote and claim to source faithfulness checker for cited PDFs.

VeriQuote reads a draft (Markdown, LaTeX, DOCX, or plain text), finds every direct quotation and every cited claim, and checks each one against the actual source document it cites. It tells you, item by item, whether the quote really appears in the source and whether the claim is supported, with the page number and the best matching passage shown as evidence.

It runs fully offline by default. The default backend is deterministic (fuzzy string alignment plus token overlap), so there is no model to download, no API key, and no data leaving your machine. Optional semantic backends (sentence embeddings, or a local LLM through Ollama) can be switched on for smarter claim scoring, but they are never required.

The gap it fills

A wave of tools checks whether a citation exists (is the cited paper real, is it in CrossRef or OpenAlex). Almost nothing checks whether the sentence you wrote is faithful to the page you cited. That second question is the one that matters when AI assisted drafts invent or misattribute quotes, and it is exactly the question VeriQuote answers. The closest open source project (a 2 star proof of concept) and the popular reference existence checkers all stop short of quote and claim to source verification. See ../OPPORTUNITIES.md for the full evidence and the market scan.

Full scope (what this project ships)

Core:

• Draft parsing for Markdown, LaTeX, DOCX, and plain text.
• Quote extraction for straight, curly, and guillemet quotation marks.
• Citation extraction for Pandoc or Markdown ([@key], with p. 12 locators), LaTeX (\cite, \citep, \citet, and friends), and numeric ([12]) styles.
• Numeric citation resolution by reading the order of a BibTeX file.
• Source loading for PDF (via PyMuPDF, page by page) and text or Markdown.
• Citation key to source file resolution with strict path containment.
• A verdict per item: supported (green), review (yellow), unsupported (red), or unresolved (the cited source could not be found).
• Page numbers and the matched evidence passage for every checked item.

Surrounding features that make it usable:

• Three report formats: a self contained HTML report, Markdown, and JSON.
• A --fail-on-red flag so it can gate a CI pipeline or a pre submission check.
• A pluggable backend system: fuzzy (default, offline), embedding (sentence-transformers), and ollama (a local LLM). Unavailable optional backends degrade gracefully to fuzzy with a warning.
• Configuration through a TOML file, environment variables, or CLI flags, with adjustable verdict thresholds.
• Safety limits on file size and item count, and a backends command to show what is available on the current machine.

Install

Requires Python 3.10 or newer (developed and tested on 3.14).

python -m venv .venv
# Windows PowerShell:  .venv\Scripts\Activate.ps1
# Git Bash / Linux / macOS:  source .venv/bin/activate
pip install -e .

For the exact pinned environment used in development:

pip install -r requirements-lock.txt
pip install -e . --no-deps

Optional extras:

pip install -e ".[semantic]"   # enables the embedding backend
pip install -e ".[dev]"        # pytest and hypothesis for the test suite

Usage

Check a draft against a folder of sources and write all three report formats:

veriquote check examples/draft.md \
  --sources examples/sources \
  --out report.html --md report.md --json report.json

You can also run it without installing the console script:

python -m veriquote check examples/draft.md -s examples/sources

Common options:

• --sources, -s DIR a directory of source files (repeatable). Files are matched to citation keys by name, for example smith2020.pdf matches [@smith2020].
• --map key=path an explicit mapping for a citation key (repeatable).
• --bib refs.bib resolve numeric citations ([12]) using the order of keys in a BibTeX file.
• --backend fuzzy|embedding|ollama choose the claim scoring backend.
• --out FILE HTML report, --md FILE Markdown report, --json FILE JSON.
• --fail-on-red exit with code 2 if any item is unsupported (useful in CI).
• --max-file-mb N reject any draft or source larger than N megabytes.

See what each item looks like:

veriquote check examples/draft.md -s examples/sources --json -   # prints JSON
veriquote backends                                               # list backends
veriquote version

Example run

The bundled example contains a verbatim quote, a fabricated quote, supported and unsupported claims, and a citation to a missing source. Running it produces:

7 items checked  supported=3 review=2 unsupported=1 unresolved=1

Open report.html to see each item color coded with its evidence passage and page number.

Configuration

Settings are resolved in this order (later wins): built in defaults, a TOML file passed with --config, environment variables, then explicit CLI flags.

TOML file (a [veriquote] table):

[veriquote]
quote_green = 95.0    # score at or above this is a verbatim match
quote_yellow = 80.0   # score at or above this is a close paraphrase (review)
claim_green = 60.0    # score at or above this means a claim is supported
claim_yellow = 40.0   # score at or above this means partial support (review)
backend = "fuzzy"
model = "qwen2.5"     # only used by the ollama backend

Environment variables:

• VERIQUOTE_BACKEND overrides the backend (fuzzy, embedding, ollama).
• VERIQUOTE_OLLAMA_HOST sets the local Ollama URL (default http://localhost:11434). Used only by the optional ollama backend.

Using the optional local LLM backend

If you run Ollama locally:

ollama pull qwen2.5
veriquote check draft.md -s sources --backend ollama --model qwen2.5

This stays local: requests go only to your Ollama host. No paid API is involved.

How to run the tests

pip install -e ".[dev]"
pytest

The suite (52 tests) generates its own PDF fixtures with PyMuPDF, so no binary files are checked in, and it runs fully offline. It covers extraction, source resolution and path safety, PDF loading, matching, the backends and their fallback behavior, the end to end pipeline, all report formats, the CLI, config validation, and the security properties below.

Architecture notes

The code is layered so each part is small and independently testable:

• models.py plain dataclasses for the domain (sources, citations, results).
• config.py configuration with clamping and validation.
• extract.py quote, citation, and claim extraction from the draft.
• pdf.py source loading (PDF and text) into the page model.
• sources.py citation key to file resolution with path containment.
• backends.py claim scoring backends (fuzzy, embedding, ollama).
• match.py the matching engine for quotes and claims.
• pipeline.py orchestration from draft to report.
• report.py JSON, Markdown, and HTML rendering.
• cli.py the Typer command line interface.

Security notes

VeriQuote processes documents that may be untrusted (a draft and the PDFs it cites), so the design treats all input as hostile.

• Path traversal is blocked. A draft drives which files are opened, via citation keys. The SourceResolver only ever returns paths that resolve (symlinks included) to a location inside an allowed base directory, and only with a whitelisted extension. Keys are validated against a strict pattern, so a crafted key such as [@../../secret] cannot escape the sources directory; it simply fails to resolve and is reported as unresolved. There is a test for this.
• Resource limits. Draft and source files larger than a configurable ceiling (default 50 MB, hard cap 500 MB) are rejected. The number of checked items is capped (default 5000, hard cap 100000). These bound memory and time on huge or malicious input.
• No catastrophic backtracking (ReDoS). Every regular expression used for extraction is linear with explicit length bounds. A test feeds adversarial input (thousands of quote and bracket characters) and asserts extraction stays fast.
• Report output is escaped. The HTML report is rendered with Jinja2 autoescaping on, so text taken from a draft or a PDF cannot inject markup or script into the report (a stored XSS style risk). There is a test asserting that <script> content is escaped.
• No secrets in code or logs. There are no API keys or credentials anywhere. The only external setting is the local Ollama host, read from the environment and never logged. Document text is never logged.
• Safe parsing only. Configuration is read with the standard library tomllib (read only, cannot execute code). The BibTeX reader is a small, bounded, regex based scanner that only extracts keys; it never evaluates anything. No eval, no pickle, no untrusted deserialization.
• Network is opt in and local. No network calls happen unless you explicitly select the ollama backend, and then only to the configured local host, with a timeout. The default run is fully offline.

License

MIT. See LICENSE.

VeriQuote is an aid, not an oracle. A green verdict means strong textual support was found, not that a human has confirmed the citation. Always review flagged items by hand before relying on them.