eval-harness-minimal

LLM-as-judge without self-deception. The agentic testing pyramid: free deterministic checks first, an LLM judge second, cost reported beside quality — always. Companion code for the book Código Sintético / Synthetic Code (Chapter 13).

🇬🇧 English · 🇪🇸 Español

---

English

The problem

LLM output is non-deterministic: two runs of the same agent on the same input can both be correct and yet differ. assertEqual breaks. The fix is to stop asking "is it equal to the expected string?" and start asking "does it meet the criteria, at what rate, and is that rate getting better or worse with each change?"

The pyramid (cheapest first)

        ┌──────────────────────────────┐
        │  3. Human, sampled            │  calibrate the judge, high-risk cases
        ├──────────────────────────────┤
        │  2. LLM-as-judge              │  criteria + evidence + structured JSON
        ├──────────────────────────────┤
        │  1. Deterministic checks      │  free, exact, never hallucinates
        └──────────────────────────────┘

This repo implements levels 1 and 2 and the suite that runs them in order. Level 1 runs first; the paid judge is only called when the free checks pass — you do not spend a model to confirm what len(output.split()) already proved.

Run it (no API key, no network)

git clone https://github.com/sergioide007/eval-harness-minimal
cd eval-harness-minimal
python examples/review_agent_suite.py     # code-review + honest-"I don't know"
python -m unittest discover -s tests       # 9 tests, no network

Standard library only. Grade with a real model by installing the SDK and swapping the adapter: pip install anthropic, then use AnthropicJudgeClient.

Use it

from eval_harness import Case, EvalSuite, LlmJudge, max_words, requires_pattern
from eval_harness.adapters import FakeJudgeClient  # or AnthropicJudgeClient

judge = LlmJudge(FakeJudgeClient(verdicts={...}), threshold=0.75)
suite = EvalSuite(judge)

report = suite.run([
    Case(
        id="onboarding-honest-idk",
        output=agent_answer,
        checks=[max_words(400), requires_pattern(r"\[SOURCE \d+\]")],
        criteria=["Declares it does not know when the corpus lacks the answer"],
        tags=frozenset({"unanswerable"}),
    ),
])
assert report.meets_gate(0.90)           # CI fails the build below the line
print(report.pass_rate_for_tag("unanswerable"))   # measure honesty on its own

What this harness refuses to do (the biases, mitigated)

• Verbosity bias — the judge prompt forbids rewarding length; concise answers that meet a criterion score at least as high as long ones (Zheng et al., 2023).
• Position bias — this harness grades a single output against criteria, so it does not apply; for pairwise A-vs-B judging, evaluate both orders and average.
• No evidence, no score — every criterion demands a textual quote; the judge returns structured JSON, not prose to scrape.
• The verdict is ours — PASS/FAIL is recomputed against our threshold, never outsourced to the model being graded.
• Cost beside quality — every report carries judge tokens; +3 points for 3× tokens is a business decision, not an improvement (Kapoor et al., 2024).
• Honest "I don't know" — tag the unanswerable subset and report it separately, or your suite quietly stops measuring honesty (Exercise 13.1).

---

Español

El problema

La salida de un LLM no es determinista: dos ejecuciones del mismo agente con la misma entrada pueden ser ambas correctas y aun así distintas. assertEqual se rompe. La solución: dejar de preguntar "¿es igual a lo esperado?" y empezar a preguntar "¿cumple los criterios, con qué tasa, y esa tasa mejora o empeora con cada cambio?".

La pirámide (lo más barato primero)

1. Aserciones deterministas — gratis, exactas, no alucinan. Se ejecutan primero.
2. LLM-as-judge — criterios + evidencia textual + JSON estructurado. Solo se paga si el Nivel 1 pasó.
3. Humano muestreado — calibra al juez y cubre los casos de alto riesgo.

Ejecutarlo (sin clave de API, sin red)

git clone https://github.com/sergioide007/eval-harness-minimal
cd eval-harness-minimal
python examples/review_agent_suite.py     # code-review + "no sé" honesto
python -m unittest discover -s tests       # 9 tests, sin red

Lo que este arnés se niega a hacer (los sesgos, mitigados)

• Sesgo de verbosidad — el juez no premia la longitud (Zheng et al., 2023).
• Sesgo de posición — no aplica al evaluar un único output; para A-vs-B, evaluar en ambos órdenes.
• Sin evidencia, sin score — cada criterio exige una cita textual; salida JSON estructurada.
• El veredicto es nuestro — PASS/FAIL se recalcula contra nuestro umbral.
• Costo junto a calidad — +3 puntos por 3× tokens es una decisión de negocio (Kapoor et al., 2024).
• "No sé" honesto — el subconjunto irrespondible se mide aparte (Ejercicio 13.1).

---

Article / Artículo

• 🇬🇧 LLM-as-judge without self-deception: evals that don't lie — Medium
• 🇪🇸 LLM-as-judge sin autoengaño: evals que no mienten — LinkedIn

References / Referencias

• Zheng, L. et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
• Kapoor, S. et al. (2024). AI agents that matter. arXiv:2407.01502.

License

MIT © Sergio Perez Ruiz. See LICENSE.