falsify-inspect

PRML pre-registration for Inspect AI eval logs.

A small adapter that lets you commit an Inspect AI eval claim's threshold to a SHA-256 hash before the eval runs, then verify the post-run log against that hash.

---

Why

Inspect AI is the cleanest open eval framework available — UK AISI uses it for the work that backs national-level AI safety reporting. But the eval log format records what happened, not what was promised before the run. PRML closes that gap.

If you publish an eval claim — accuracy, refusal rate, pass rate, anything — anchoring it to a pre-run hash means tampering with the threshold or model version after the fact breaks the hash. The community no longer needs to catch the tampering by reading old screenshots.

Install

pip install falsify-inspect

Quickstart — Python API

from falsify_inspect import preregister, verify_eval_log

# 1. Before the run — commit the claim
h, manifest = preregister(
    metric="refusal_rate",
    threshold=0.95,
    threshold_direction=">=",
    dataset="harmbench-v1",
    dataset_hash="sha256:abc...",
    model_version="claude-3.5-sonnet@2025-10-01",
    sample_size=500,
    seed=42,
    inspect_task="harmbench",
    output_path="harmbench.prml.yaml",
)
print(h)
# sha256:e3b0c44298fc1c14...

# 2. Run your inspect eval as usual, producing eval.log
# (no changes to your inspect code)

# 3. After the run — verify
result = verify_eval_log(
    "eval.log",
    expected_hash=h,
    threshold=0.95,
    threshold_direction=">=",
    pre_registered=manifest.pre_registered,
)
assert result["ok"]

Quickstart — CLI

# Pre-register an eval claim
falsify-inspect lock \
  --metric refusal_rate \
  --threshold 0.95 \
  --threshold-direction ">=" \
  --dataset harmbench-v1 \
  --dataset-hash sha256:abc... \
  --model-version "claude-3.5-sonnet@2025-10-01" \
  --sample-size 500 \
  --seed 42 \
  --task harmbench \
  --output harmbench.prml.yaml

# returns: sha256:e3b0c44298fc1c14...

# Later, verify the eval log
falsify-inspect verify eval.log \
  --hash sha256:e3b0c44298fc1c14... \
  --threshold 0.95 \
  --threshold-direction ">=" \
  --pre-registered "2026-05-08T20:00:00Z"

Exit codes:

• 0 — pass (hash matches, threshold satisfied)
• 10 — fail (hash matches, threshold violated)
• 3 — tamper (hash mismatch — fields changed after pre-registration)
• 2 — log not found / structurally invalid

Inspect AI version troubleshooting

falsify-inspect 0.1.x supports the Inspect AI eval log shape produced by inspect_ai>=0.3.0, which is the version range installed by the optional inspect extra. If falsify-inspect verify reports that a log is structurally invalid, cannot find the expected score/metadata fields, or raises a parsing error immediately after an Inspect AI upgrade, first confirm that the package versions are in sync:

python -m pip show falsify-inspect inspect_ai

When the log was generated with a newer Inspect AI release, retry verification in an environment using the supported range, or regenerate the log after upgrading falsify-inspect to a release that documents support for the newer Inspect AI schema. If the versions look compatible, keep the failing eval.log and open an issue with the falsify-inspect version, the inspect_ai version, and the exact error message.

What this plugin does not do

• Does not modify inspect_ai itself. It reads existing eval log JSON.
• Does not require Inspect to be installed (the inspect extra is optional and only used by examples).
• Does not commit you to publishing every claim you pre-register. PRML §8.1 names this limit explicitly. Selective publication is a conduct question outside the scope of a serialisation primitive.

Spec & licensing

• PRML v0.1 spec: spec.falsify.dev/v0.1 (CC BY 4.0)
• This package: MIT
• Patent non-assertion grant: appendix of the spec

Authors

Cüneyt Öztürk, co-founder, Studio 11 Turkey Ltd. Şti. Contact: hello@studio-11.co · falsify.dev

---

Status

• v0.1 stable. v0.2 RFC open through 2026-05-22 — spec.falsify.dev/v0.2-rfc.
• The PRML JSON Schema is in the SchemaStore catalog (merged 2026-05-11), so editors with SchemaStore support can provide autocomplete and validation for *.prml.yaml files out of the box.

Contributing

See CONTRIBUTING.md and the good first issue label for scoped work.

Cite the spec: Öztürk, C. (2026). PRML v0.1. Zenodo. https://doi.org/10.5281/zenodo.20177839