Verified by Probity (github.com/probityai/probity): latest passing run
Test an agent-evidence verifier against saved records with known outcomes. The corpora cover valid records, altered bytes, missing evidence and open specification interpretations.
The recording is demo/four-arms.sh from a fresh clone; vhs demo/four-arms.tape re-records it.
Add it to a project with uv add agent-evidence-vectors==0.26.0, then pip install agent-evidence-vectors==0.26.0. The probityai command line is planned, not yet released: uv add probityai, then pip install probityai.
Run every vector in the default corpus through the bundled reference verifier. You need Python 3.11+ and uv; there is nothing to clone or build, and a cold run takes about six seconds:
uvx agent-evidence-vectors==0.26.0At v0.26.0 the default corpus is suiteRevision 32, and the run prints one row per vector and ends with:
totals: 288 vectors, 288 pass, 0 fail; report written to conformance-report.json
Now run the same corpus against a verifier other than the reference. The Go reference verifier installs with Go 1.26+ and Go's executable directory on your PATH; put your own command in its place:
go install github.com/probityai/agent-evidence-vectors/cmd/aee-verify@v0.26.0
uvx agent-evidence-vectors==0.26.0 --verifier "aee-verify -validity-only -json"Open conformance-report.json for the per-vector results. A complete external run has rail: external, verifier.vectorsExecuted equal to totals.vectors, and zero totals.suiteRefusals. The report separates conformance from reason-code agreement. --list-corpora names the other shipped corpora and --corpus <name> selects one.
The release is v0.26.0 on GitHub and PyPI. To run from a reviewed source checkout instead, follow the source build. Installation routes and release identity covers package availability and signature checks.
The harness calls your executable with a vector-file path. Return a verdict through the exit status and a JSON object on the last line of stdout. The verifier contract covers output fields, key policy and comparison rules.
Use the GitHub Action to run the same check in your build and retain its report. The CI guide gives a workflow step and the available inputs.
- uses: probityai/agent-evidence-vectors@v0.22.0
with:
verifier: VERIFIER_COMMAND --jsonReplace VERIFIER_COMMAND with the command that runs your verifier.
A runtime that retries tool calls can run the uncertain-effect scenarios: a write that commits and then loses its reply must not run twice.
- uses: probityai/agent-evidence-vectors@v0.30.0
with:
corpus: vectors-uncertain-effect
runtime: ADAPTER_COMMANDagmi, Rul1an/aee-checker and MoltyCel/moltrust-api run a corpus from this repository in their own CI at a pinned release or commit, and this repository's CI runs agmi's anchored-chain verifier back against vectors-anchored-chain. The verifier table gives each one's corpus, pin and CI run.
| I want to | Read |
|---|---|
| Start a project with this Action already in CI | evidence-starter template |
| Pick a format and corpus | Corpus guide |
| Implement the verifier interface | Verifier contract |
| Understand what the fixtures test | Corpus measurements |
| Check release bytes and signatures | Release verification |
| Report a run or disagreement | Run reporting |
| Check a W3C intake record | W3C intake checker |
| See who runs and uses the corpora | Independent runs, Adopters |
Each corpus manifest records its source revision, digest and expected outcomes. DISTRIBUTION.md lists the shipped corpora and installation routes.
For fixture coverage, see Corpus measurements. To check signed release bytes, use Release verification.
Source maps: repository layout, corpus readers and witness attestor.
Open an issue with the corpus revision, verifier build and report. For patches, start with CONTRIBUTING.md. For citations, use the citation guide.
Need to check a supplied claim against saved evidence? See Probity Verify. Browse the Probity projects for records, vocabulary and admission tools.
Apache-2.0. LICENSE.