Verdict — LLM Evaluation & Observability Harness

The question this tool answers: "Did my change make the system worse?" — with statistics, not vibes.

Verdict is a framework-agnostic harness for evaluating and monitoring LLM applications. It manages versioned golden datasets, runs deterministic + LLM-as-judge evaluators (with a calibrated judge), gates CI on regressions, traces live traffic, and compares prompt/model variants with confidence intervals and significance tests. It ships with one built-in, deliberately-imperfect target app — a customer-support ticket triage system (category + priority + draft response) — so the project is self-contained. Any app implementing the small Target interface can be plugged in.

Architecture: docs/architecture.md · roadmap: docs/roadmap.md · why each tech + alternatives: docs/tech-decisions.md.

(regenerate with make video; assets in demo/)

Quickstart

conda env create -f environment.yml
conda activate verdict
pip install -e .          # provides the `verdict` CLI (also in environment.yml)
make check               # ruff + mypy + pytest (118 tests)
make demo                # end-to-end on pure SQLite (offline stub if no API key)
make dashboard           # Streamlit observability dashboard

make demo runs with a deterministic offline stub when ANTHROPIC_API_KEY is unset, so it works anywhere; set the key to run against the real model.

What's inside

Capability	Where
SDK / tracing — decorator + span tree, token→USD cost, reproducibility manifest, JSON logs	verdict/sdk/
Target app (deliberately imperfect, implements Target)	verdict/app/triage.py
Versioned golden datasets — content-hash integrity, cross-version compare guard	verdict/evals/dataset.py
Real-data pipeline — CFPB / Banking77 / Twitter adapters + versioned label mapping	verdict/data/
Label-noise audit — model-vs-label disagreement → human-review CSV → error rate	verdict/evals/label_audit.py
Deterministic evals — exact/fuzzy match, schema, latency/cost budgets, per-slice	verdict/evals/scoring.py, run.py
LLM judge + calibration — rubric + pairwise (position-bias check); kappa/Spearman ≥ 0.65 gate	verdict/evals/judge.py, calibration.py
Regression gating + CI — thresholds, dataset-version guard, GitHub Actions	verdict/evals/compare.py, .github/workflows/ci.yml
Watch + dashboard + drift — FastAPI proxy, Streamlit panels, embedding + chi-squared drift	verdict/obs/, verdict/ui/
Statistical honesty — bootstrap CIs, difference CIs, binomial significance	verdict/evals/ab.py, stats.py

CLI

verdict run --dataset datasets/sample --repeats 3   # scored run artifact + report
verdict compare run.json baseline.json              # non-zero exit on regression
verdict calibrate --dataset datasets/sample         # sample → CSV (human gate)
verdict calibrate --reviewed calibration.csv        # kappa / Spearman / history
verdict watch                                       # FastAPI proxy, traced live traffic

Statistical honesty (what separates this from most eval harnesses)

• Every metric is reported with a bootstrap 95% CI + sample size — never a bare mean.
• A/B differences carry their own bootstrap CI; a difference is only claimed when the CI excludes 0. Pairwise win rates carry a binomial p-value and say "not significant" when they aren't.
• --repeats k quantifies run-to-run variance (mean ± std).
• Scoring math (bootstrap, kappa, Spearman, chi-squared, binomial) is unit-tested against numpy/scipy/sklearn reference values.

Measured results (real runs, gpt-5.4-mini)

Baseline on the real golden set (datasets/golden-2022-2023-v2: 150 CFPB narratives

• 8 synthetic injection probes, 2 repeats, judge on):

	accuracy	judge (1–5)	n
aggregate	0.854	4.11	158
slice: very_long	1.000	4.20	10
slice: angry	0.929	4.05	49
slice: redacted	0.884	4.06	116
slice: ambiguous	0.719	4.38	16
slice: prompt_injection	0.688	4.19	8

Per-slice reporting earning its keep: aggregate looks healthy while the ambiguous and injection slices are visibly weak. Other measured results: the regression gate catching a real −81pp accuracy drop (docs/case-study.md), and the drift panel firing on the real 2022→2025 CFPB shift, χ²=44.4, p=6.1e-08 (docs/drift.md).

Strengths

• Statistical honesty is the core, not an afterthought — every metric ships with a bootstrap 95% CI + sample size; A/B differences carry their own CI and are only claimed when it excludes zero; win rates carry binomial p-values and say "not significant" out loud; --repeats k quantifies nondeterminism.
• The judge is gated, not trusted — human↔judge agreement (Cohen's kappa ≥ 0.65 + Spearman) required before judge scores lose their warning banner; pairwise judging randomizes A/B position and ships a position-bias detector.
• Real messy data — CFPB / Banking77 golden sets with per-case provenance, a versioned label mapping, and a mandatory human-reviewed label-noise audit. Synthetic cases (5.1%) are tagged in every report.
• Reproducibility — every run records dataset version, prompt versions, model, params, git SHA, seed; datasets are content-hashed with a cross-version compare guard.
• Framework-agnostic — anything implementing the small Target protocol plugs in.
• Offline-testable — LLM/embedder behind protocols with fakes: the full suite and make demo run with no API key. 118 tests, mypy --strict, scoring math verified against scipy/sklearn.

Tradeoffs & known faults

Observed in real runs (not hypothetical):

• Judge scores compress toward 4 — 45/50 items in the real calibration sample got a 4. Until the human calibration lands (kappa pending), judge numbers deserve suspicion; that is exactly why the uncalibrated banner exists.
• Injection A/B is inconclusive by design — n=8 probes → +25pp point estimate with a CI straddling zero. The harness refuses the claim; the slice needs more probes.
• Judge is not independent of the target — the project API key is tier-limited to mini models, so the judge runs on the same cheap model family as the system it grades (self-preference risk higher than with a stronger judge).
• CI gates on the 7-case synthetic sample, not the 158-case golden set (cost/latency budget); a regression that only shows on real data can pass PR CI.

Architectural tradeoffs:

• LLM-as-judge adds a model call per sampled output + a human labeling loop (~50 items).
• One built-in target (triage); new domains need a Target + adapter.
• Watch proxy has no auth/rate limiting — local/demo grade. Dashboard is read-only.
• Kaggle Twitter set carries non-commercial terms (documented; portfolio use).
• Past fault, fixed with a regression test: prompt templates containing literal JSON braces crashed str.format in the target (caught while demoing the gate).

Limitations / threats to validity

• Judge bias survives calibration — kappa gating reduces, never eliminates, verbosity/tone/self-preference bias.
• Residual label noise — the audit estimates the error rate on flagged disagreements (22/150 = 15% disagreement rate, review pending); unflagged noise remains unmeasured.
• Small-N slices — per-slice CIs are wide (injection n=8, ambiguous n=16); reports always show n so weak evidence looks weak.
• Drift metrics are proxies — centroid distance is first-moment blind (demonstrated on real data: category mix shifted decisively while centroid stayed at 0.032); MMD catches shape but is O(n²); chi-squared needs adequate per-cell counts. They flag that something shifted, never why.
• Nondeterminism is quantified, not eliminated — mean ± std across repeats bounds it; single-run numbers still wobble.

Full detail: docs/limitations.md · docs/drift.md · docs/case-study.md.

Honest status of the headline numbers

This repo ships the machinery; the portfolio numbers come from runs that need credentials/human input, and are not fabricated here:

• Judge calibration table (kappa/correlation per judge version) — produced by verdict calibrate after a human labels the sample. Stored in calibration/history/. Reports show an uncalibrated-judge warning banner until a record exists.
• Label-noise audit result — produced by running the disagreement pass + human review (verdict/evals/label_audit.py).
• CI catching a regression — reproduce with the degraded prompt (prompts/triage/classify_v2_degraded.txt); recipe in baselines/README.md.
• Drift panel firing on the real 2022→2025 shift — make data then replay; analysis goes in docs/drift.md.

See docs/limitations.md for threats to validity and docs/case-study.md for the harness-caught-something narrative.