Schema: evaluations, datasets, metrics, and release gates

[rajanbor]

Goal

Define library/schemas/eval.schema.json for reproducible AI quality checks.

Human contract

Explain objective, dataset source, cases, rubric, deterministic checks, judge configuration, metrics, thresholds, sampling, known limitations, and release-gate behavior. Include one deterministic and one LLM-judge example.

Machine contract

Type dataset and prompt references, metric direction, thresholds, repetitions, seed where applicable, judge independence, and pass/fail aggregation. Separate secrets and runtime credentials from committed contracts.

Acceptance criteria

• Schema, guide, two valid examples, and invalid fixtures exist.
• Every metric has an unambiguous direction and threshold.
• Judge-based evals require a rubric and calibration notes.
• Missing datasets/prompts are detected before execution.
• CI can produce a deterministic machine-readable result envelope.

Depends on document and prompt schemas.