A verifier datasheet is a one-page audit record for any mechanism that turns model behavior into a verdict: a grader in an RL environment, an LLM-judge rubric, a failure detector, a reward function. Datasets got datasheets (Gebru et al., 2018) and models got cards (Mitchell et al., 2018). Verifiers, the component that decides what gets reinforced and what gets reported, mostly ship with a headline metric and no chain of custody.
This repository contains:
- TEMPLATE.md: the datasheet template. Identity, label lineage, agreement statistics, adjudication record, calibration record, known limits.
- datasheets/derailment-wildchat.md: a filled-in worked example for a task-derailment LLM judge measured on WildChat, including a three-version labelling history and a contested-label adjudication in which 7 of 9 positives flipped on full-input review.
- data/: sanitized exports backing every number in the worked example. Per-trace panel verdicts (97 rows, three vendor families), the contested-label adjudication record with lineage fields, and the aggregate agreement artifact. Conversation text is not redistributed; rows join to the public WildChat dataset via
source_trace_id. - scripts/compute_agreement.py: recomputes the pairwise agreement table (raw agreement, positive specific agreement, Cohen's kappa) from the verdict export. No dependencies.
python scripts/compute_agreement.py
Two claims, both load-bearing for anyone buying or building RL environments and evals:
- A high agreement number is not evidence by itself. The worked example shows raw cross-vendor agreement of 0.96 to 0.98 alongside positive specific agreement of 0.00, on the same slice, at 2 percent prevalence. Which number you publish decides whether the reader is informed or misled.
- Inter-judge agreement cannot catch a defect all judges share. In the adjudication record, positives that were artifacts of input truncation survived the three-vendor panel vote, in one case unanimously, because every judge saw the same truncated evidence. The fix was full-input review plus a doctrine ruling, and the datasheet records both.
The format is deliberately boring. Every row is something a skeptical reviewer would ask for anyway; writing it down first is cheaper.
Extracted from the calibration infrastructure of Pisama, a failure-detection platform for multi-agent LLM systems. The numbers are real measurements from the production labelling lane, including the unflattering ones. That is the point.
Code and documentation: MIT. The verdict exports describe, but do not contain, WildChat conversations; WildChat itself is distributed by AI2 under its own license terms.