Version: 1.0.0-rc.4 | License: Apache 2.0
Adopted by: Inspect AI — merged to the official community extensions list, PR #4797, August 2026. Approved with a clean CI run at TruLens — PR #2697, a full to_openeval()/from_openeval() module, 17/17 tests passing, awaiting merge. 32 framework adapters shipped and independently tested — including Agenta (built at the explicit request of the Agenta-AI maintainer, Agenta-AI/agenta#6222), Parea AI (a first-time external contributor's PR #19, merged after review and an independent re-run of its test suite), Hugging Face evaluate, EleutherAI lm-evaluation-harness, Hugging Face lighteval (the library behind the HF Open LLM Leaderboard), OpenCompass, FinanceBench (a real benchmark dataset, not a live SDK), Athina, DeepEval, and Galileo — plus a Literal AI adapter in review right now from another new contributor (PR #25) — see Framework Adapters below.
EvalPort is an open specification for portable LLM evaluation test cases, graders, suites, and results. It enables evaluation datasets to be shared across frameworks (DeepEval, Promptfoo, Ragas, Inspect AI, LangSmith, Braintrust, OpenAI Evals, MLflow) without loss of semantic fidelity.
# Install CLI
npm install -g evalport-cli
# Create an eval suite
openeval init my-eval-suite
# Validate
openeval validate my-eval-suite.json
# Convert from Promptfoo
openeval convert promptfoo openeval config.json output.json
# Run a suite against a real provider — always dry-run first to see estimated cost
openeval run examples/basic-suite.json --provider openai --model gpt-4o-mini --dry-run
openeval run examples/basic-suite.json --provider openai --model gpt-4o-mini --output results.json
# Python SDK
pip install evalport-sdkfrom openeval.validate import validate_suite
result = validate_suite({
"version": "1.0.0",
"id": "my_suite",
"graders": [{"id": "gr1", "type": "exact_match"}],
"test_cases": [{"id": "tc1", "input": "Hello", "expected_output": "Hi", "graders": ["gr1"]}]
})
print(result.valid) # TrueEvalPort is pre-1.0 and actively shaped by outside contributors — the fastest ways in, from least to most involved:
- Have an opinion on an open design question?
spec/SPEC.mdtracks what's genuinely unresolved. Right now that's Discussion #22 — howResultSetshould represent repeated-attempt evals (num_repetitions, epochs), raised by a real downstream consumer, not an internal critique. Four earlier questions on the same list (suite/result signing, a formal conformance test suite, resuming interrupted runs, and whetherllm_judgeinjection mitigations should be mandatory) already went through this exact process and landed as shipped spec changes — the table shows how each one resolved. No prior EvalPort contribution required, just a considered opinion. - Want to ship a framework adapter? Issue #6 is the map.
adapters/autogen-openeval-adapteris the reference shape to copy:to_openeval(),from_openeval(), tests against the real validator, a README. - Want to propose a spec change? Open a Discussion titled
[Spec Change] <what and why>— the full process (comment period, what "consensus" means, when the spec lead's sign-off is required) is inspec/SPEC.md's Governance section. - How people actually become collaborators: ship something real and tested, engage substantively, get invited — see
CONTRIBUTORS.mdfor who's done that so far. It's not gatekept; a merged, tested PR is what clears the bar.
Full details in the Contributing Guide.
openeval/
├── spec/
│ ├── SPEC.md # Full specification
│ ├── ADOPTION.md # Adoption strategy
│ ├── CRITIQUE.md # Self-critique and hostile review
│ ├── schemas/ # JSON Schemas (4 files)
│ ├── examples/ # Example suites and conversions
├── sdk/
│ ├── typescript/ # evalport-sdk (npm)
│ └── python/ # openeval (PyPI)
├── cli/ # evalport-cli
├── api/ # Example REST API server
├── docs/
│ ├── getting-started/ # 5-minute quickstart
│ ├── grader-reference/ # All 11 grader types
│ ├── migration-guides/ # Promptfoo, DeepEval, Inspect AI
│ ├── api/ # REST API docs
│ ├── blog/ # Launch posts
│ └── landing-page.html # Landing page
├── examples/ # Example eval suites
├── adapters/ # Standalone to_openeval()/from_openeval() packages per framework
├── benchmarks/ # 14 public benchmarks converted to validated EvalPort suites
├── .github/ # CI, CONTRIBUTING, issue templates
├── LICENSE # Apache 2.0
└── README.md # This file
evalport run executes an EvalPort suite against a real model provider and produces a spec-valid, self-validated ResultSet — no separate harness, no glue code. It's the CLI's headline command:
openeval run suite.json --provider openai --model gpt-4o-mini --dry-run # estimate cost first, spend nothing
openeval run suite.json --provider anthropic --model claude-3-5-sonnet-20241022 --output results.json- Two providers out of the box — OpenAI and Anthropic — plus any OpenAI-compatible endpoint (local inference servers, proxies, other vendors) via
--api-base. - Tier 1 graders run locally, with zero external dependencies:
exact_match,contains,regex,json_schema(a hand-written draft-07-ish validator),json_path(a hand-written JSONPath subset evaluator). - Tier 2 graders call an API:
llm_judge/model graded, andsemantic_similarity(cosine similarity over embeddings — always via an OpenAI-compatible endpoint, independent of--provider, since Anthropic has no public embeddings API). - Unsupported grader types clean-skip, per the spec's "Custom grader handling" rule (
code,human,customare recorded asskippedwithmetadata.skip_reason: "unsupported_grader_type"— the run never aborts because of one grader it doesn't know how to execute). --dry-runestimates cost before spending anything — token and USD estimates per test case and in total, with warnings when a model's pricing isn't in the known table. Always run this first and get sign-off on the estimate before running for real.- Retries with backoff on retryable provider errors (HTTP 429/5xx); non-retryable errors (bad auth, malformed request) fail fast instead of repeating.
--parallel <n>for concurrent test cases,--limit <n>to run a subset,--output <path>to write results incrementally as cases complete (so a long run's progress survives an interruption).- Every
ResultSetit produces is validated against the SDK's ownvalidateResultSet()before being written —evalport runrefuses to emit output that fails its own spec.
See cli/README.md for the full flag reference.
| Type | Description |
|---|---|
exact_match |
String equality |
contains |
Substring check |
regex |
Regex match |
semantic_similarity |
Embedding cosine similarity |
llm_judge |
LLM-as-judge with prompt template |
json_schema |
JSON Schema validation |
json_path |
JSONPath extraction + comparison |
code |
Custom grading function (sandboxed) |
human |
Human review |
model graded |
Alias for llm_judge (OpenAI Evals compat) |
custom |
Framework-specific handler |
| From | Status |
|---|---|
| Promptfoo → EvalPort | ✅ CLI + SDK |
| DeepEval → EvalPort | ✅ Python SDK |
| Inspect AI → EvalPort | ✅ Python SDK |
| OpenAI Evals → EvalPort | ✅ Python SDK |
benchmarks/ converts 14 well-known public benchmarks — GSM8K, ARC, BoolQ, HellaSwag, WinoGrande, CommonsenseQA, PIQA, TruthfulQA, MMLU, HumanEval, MBPP, SQuAD 2.0, DROP, and BIG-Bench Hard — into 22 individually-valid EvalPort suites (8,012 test cases total), every one of them passing the real validate_suite() validator in CI. Every benchmark's license was independently verified before inclusion; see benchmarks/LICENSES.md for the full attribution table. Start with benchmarks/README.md for the full index and quickstart.
adapters/ has standalone to_openeval()/from_openeval() packages for converting real evaluation results to and from EvalPort, one per framework, each independently installable and tested against the real validator:
- Getting Started
- Grader Type Reference
- Migration Guides
- REST API
- Full Specification
- Adoption Strategy
- Hostile Critique
- Contributing Guide
- Governance & the RFC process
- Open Design Questions — live Discussions on unresolved spec questions, no prior contribution required
- GitHub Discussions
- Contributors
- Launch Blog Post
- Landing Page
Apache 2.0 — see LICENSE