Open tools that check claims about AI agents against evidence you pinned: an offline verifier today, with the observer joining this repository. The standard layer (corpus, vocabulary, admission rules and crates) lives in https://github.com/probityai/agent-evidence-vectors.
It's for auditors, incident reviewers and relying parties who need a decision they can replay, and for tool authors who want their records checked the same way.
Install the released verifier: uv add probity-verify==0.2.4, then pip install probity-verify==0.2.4.
Verified by Probity (github.com/probityai/probity). The probityai command line is released: uv add probityai==0.1.0, then pip install probityai==0.1.0.
Check a sample claim with the released package. You need Python 3.11 or newer and uv; there is nothing to clone:
samples="$(uvx --from probity-verify==0.2.4 python -c 'import importlib.resources as r; print(r.files("probity_verify") / "samples")')"
uvx probity-verify==0.2.4 "$samples/case.json" --policy "$samples/policy-june.json"It prints a decision packet that opens:
Probity decision packet v1
Case: "source-coverage-demo"
Claim: source_text_coverage/v1
Decision: contradicted
Reason: selected_span_missing_from_record
A selected passage is in the source capture but missing from the stored report. Swap in policy-april.json for not_established, or case-complete.json with policy-complete.json for supported.
| member | what it is | released |
|---|---|---|
| packages/probity-verify | offline verifier: supported, contradicted or not_established for a claim against pinned bytes |
0.2.2 on PyPI |
| packages/observer | records what an agent run did into pinned evidence the verifier can check | 0.1.1 on PyPI (agent-evidence-observer) |
| packages/ratchet | signed, replayable receipts for improving an agent against held-out cases | 0.3.2 on PyPI (probity-ratchet) |
| packages/probityai | the probityai command; probityai verify today, each member adds its own subcommand |
0.1.0 on PyPI (probityai) |
uv sync --all-packages --all-extras
uv run python -m pytest| page | read it for |
|---|---|
| probity-verify | the verifier, its adapters and its recipes |
| Repository layout | the workspace, the probityai command seam, CI and releases |
| Agent instructions | commands to run before a change |
| Contributing | reporting a problem and opening a pull request |
Apache-2.0, per directory: each member carries its own LICENSE.