evalgate

A regression gate for LLM systems. evalgate scores an LLM system's answer quality against a committed baseline and fails CI when a change degrades quality. Wire it into your pipeline and a pull request that lowers answer quality fails the build.

What & why

Most "evals" are a notebook someone runs once. evalgate is the opposite: a gate that runs on every change and refuses to let quality silently rot. It is built on a strict two-artifact split:

1. The harness (evalgate/) — system-agnostic core. It knows only an adapter interface, five metrics, a regression gate, reporting, and a CLI. It never imports a system-under-test.
2. Adapters (adapters/) — the only place a real system plugs in. A consumer points the harness at their own system by implementing one method, SUTAdapter.query(). ARA is the first reference integration.

The dataset owns the reference/ground-truth (GoldenSample); the adapter owns the system output (EvalSample); the runner merges them into a ScoredSample. That split exists because the RAGAS context metrics are defined relative to a reference the adapter can't provide.

Quickstart

uv sync
export ANTHROPIC_API_KEY=...        # the judge needs this

# 1. Pin the baseline from the frozen fixtures (one time, reviewed commit):
uv run evalgate baseline-update --adapter fixture \
  --fixture-dir fixtures/ara_recorded --baseline baselines/baseline.json

# 2. Run the gate (exits non-zero on regression or judge mismatch):
uv run evalgate gate --adapter fixture \
  --fixture-dir fixtures/ara_recorded \
  --baseline baselines/baseline.json --report report.json

The offline unit suite needs no API key:

uv run pytest -m "not live" -q

The five metrics

Scope is fixed at five metrics in v0 — higher is better for all, scores are floats in [0, 1].

Metric	Library	What it catches
Faithfulness	DeepEval	Answer hallucinates claims not supported by the retrieved context.
Answer Relevancy	DeepEval	Answer drifts off the question that was asked.
Context Precision	RAGAS	Retriever pulls in irrelevant chunks (low signal-to-noise).
Context Recall	RAGAS	Retriever misses context needed to answer (relative to the reference).
Citation Grounding	custom, deterministic	Claimed citations are absent from the retrieved context. No judge — pure string check.

The judge

Quality is scored by an LLM judge, default claude-sonnet-4-6 at temperature 0.

• The judge is a config value, never hardcoded (evalgate.yaml → judge:), and the full judge string (provider:model@version) is recorded into every report and baseline.
• Judge-consistency gate: the gate refuses to compare a run to a baseline produced under a different judge string. A silent judge upgrade must not be allowed to masquerade as a quality regression — re-pin the baseline deliberately instead.
• Judge calls are cached on (sample_id, metric, judge_version, prompt) and retried with exponential backoff, so a transient 429/timeout never fails a build for a non-quality reason.
• Self-preference bias: ARA generates answers with Claude and the default judge is also Claude, so the judge may mildly favor same-family outputs. This is acknowledged, not hidden. The judge is selectable via the provider/model config keys, so a cross-family judge can be pinned when bias matters for your system.

Measured tolerance

The gate's regression band (tolerance) must sit above the judge's own run-to-run noise, otherwise normal judge jitter would trip the gate. The noise is measured, not guessed:

export ANTHROPIC_API_KEY=...
uv run python scripts/measure_judge_variance.py 5   # 5 cache-disabled runs over the fixtures

The script prints per-metric mean, stdev, and a suggested_tolerance >= 2*stdev.

Measured result (judge anthropic:claude-sonnet-4-6@20260501, 5 runs over the frozen fixtures/ara_recorded set):

Metric	Baseline (mean)	Measured stdev	Threshold (floor)	Tolerance
faithfulness	1.00	0.000	0.80	0.10
answer_relevancy	1.00	0.000	0.80	0.10
context_precision	0.60	0.000	0.45	0.10
context_recall	0.85	0.000	0.70	0.10
citation_grounding	1.00	0.000	0.90	0.02

Judge noise measured at ±0.000 — at temperature 0 the judge is deterministic on this fixture set; no verdict flipped across 5 runs. A literal tolerance of 2*stdev = 0 would be reckless, though: with 10 samples, a single future sample's verdict flipping moves an aggregate by up to 0.10. So tolerance is set above the measured noise and sized to absorb one per-sample flip (0.10 for the judged metrics; 0.02 for citation grounding, which is deterministic and truly stdev 0). Thresholds are absolute floors set below the measured baseline.

Why context_precision is 0.60, not ~1.0 (and why that's correct, not a bug): RAGAS LLMContextPrecisionWithReference judges each retrieved context individually against the whole reference answer. On multi-hop questions whose answer combines two facts (e.g. which body approves the budget and where it meets), no single context justifies the entire answer, so each scores 0 — dragging precision down. The edge-case probes with correctly-empty retrieval also score 0. This is the metric discriminating exactly as intended on a deliberately-hard dataset; the baseline captures it honestly rather than hiding it behind single-fact questions.

Adding your own system

Implement one method. That's the whole integration surface:

class SUTAdapter(Protocol):
    name: str
    def query(self, question: str) -> EvalSample: ...   # answer, retrieved_contexts, citations

adapters/ara.py is the worked example — a ~20-line HTTP adapter that POSTs the question to a service and maps the JSON response into an EvalSample:

class ARAAdapter:
    name = "ara"
    def query(self, question: str) -> EvalSample:
        resp = self._http().post(f"{self._base}/query", json={"question": question})
        resp.raise_for_status()
        data = resp.json()
        return EvalSample(answer=data["answer"],
                          retrieved_contexts=data["retrieved_contexts"],
                          citations=data["citations"])

Point the CLI at it with --adapter ara --ara-url <url>. The harness core never imports your system; only the adapter does.

Dataset provenance

datasets/golden.yaml is the frozen reference set. Each entry carries a provenance field explaining why it is in the set. Samples are selected for category coverage — factual, multi-hop, synthesis, and edge-case (out-of-scope / ambiguous probes that catch systems which hallucinate when they should abstain).

Editing golden.yaml changes the contract being tested, so it is reviewed as code, not edited casually. datasets/degraded.yaml plus fixtures/degraded/ hold deliberately-bad outputs whose citations are absent from their context; tests/test_degraded_gate.py asserts the gate fails on them, proving the gate actually fires.

Baseline guardrail

baselines/baseline.json holds the per-metric scores and the judge string they were produced under. It is updated only by an intentional, reviewed commit (evalgate baseline update). The baseline.json diff is the social control: a reviewer sees exactly which metric moved and by how much before a new bar is accepted. Runs above baseline are flagged "improved" and pass, but the baseline is never auto-repinned.

CI

Two-speed, so the common path stays fast and cheap:

• eval-pr.yml — runs on every pull request. Offline unit tests (no key) → quality gate over the frozen ARA fixtures with the live judge (cached). This is the workflow that fails the build. Needs the ANTHROPIC_API_KEY secret.
• eval-nightly.yml — schedule + manual dispatch. Runs the live ARA HTTP adapter end-to-end against the full dataset. Needs ANTHROPIC_API_KEY and ARA_URL secrets.

Proof: the gate blocks a quality regression

The mechanism was validated end-to-end with a deliberate demo PR that changed one answer to cite a source absent from its retrieved context — a pure answer-quality regression, with no failing tests and no broken code. citation_grounding dropped 1.00 → 0.90 and the gate turned the build red:

✓ Unit tests (no API key needed)          # code is fine — tests pass
✗ Quality gate (frozen fixtures, live judge)
    FAIL: citation_grounding 0.900 vs baseline 1.000 (fail_regression). 1/5 metrics regressed.
    ##[error]Process completed with exit code 1.

The failed run is preserved here: Actions run #28940428062. This is the whole thesis made concrete: a change a normal pipeline would wave through (compiles, tests green) is caught because it lowered answer quality against the committed baseline.

Testing this end-to-end also surfaced a real CI bug: the gate's non-zero exit was initially swallowed by a pipe to tee under bash -e (no pipefail), so the first run passed green despite printing FAIL. Fixed with set -o pipefail; the re-run failed correctly. Had the gate only ever been run locally, it would have shipped silently ineffective.