evalplane

A distributed evaluation platform for coding agents. Register a benchmark, run it across a worker fleet, and compare experiments with lineage and statistical honesty.

It exists because three of my agent-research repos independently reinvented the same two things, running an eval and recording a run, and none of them agreed on what a run was.

Repo	Run identity	Runs
review-agent-lab	runs/{iteration}-{ts}.jsonl	225
sidekick-harness	results/{task}__{config}@{driver}__{unix}.json	19
rl-environments	tasks/{task}/runs/{stamp}-v{n}-{model}/	37

Each carried roughly the right fields in a different shape. Only one hashed its config. Nothing could answer "every run of Fable across all three, by cost per solved task" without a bespoke script. All 281 runs now live in one queryable store, and the platform is the extracted common layer rather than a fourth sibling.

What it does that a results tracker doesn't

It refuses invalid comparisons. Two experiments are comparable only if they ran the same benchmark, at the same version, over the same case set.

$ evalplane compare partial-refund-cent-loss@opus@v3 partial-refund-cent-loss@opus@v4
  ✗  not comparable
    - benchmark version differs: partial-refund-cent-loss@3 vs @4

Those two sat in adjacent directories looking directly comparable. Raw SQL reports the difference between those two happily. The guard names the reason and shows no number, because a delta printed beside a warning still gets quoted.

It says when a difference is too small to believe. The guard answers whether two results may be compared. This answers the more commonly botched question, whether the difference between them is real.

$ evalplane compare I0-validate I2a-intent
  ✓  comparable: review-lab@1, 13 case(s), n=1

  paired precision delta  -0.038   CI [-0.115, +0.000]   within noise
      an effect this size needs ~50 paired case(s) to resolve; you have 13

A paired bootstrap over the shared cases, seeded from the experiment ids so the same query always returns the same interval.

The interval is drawn as a span against a zero rule, so whether a difference clears zero is something you see rather than read.

It compares on precision without being asked because review-lab declares a default_metric. Its recorded score is recall, which sits at 1.0 almost everywhere since the reviewer catches nearly every in-diff bug, so comparing on it would report no difference no matter what changed. A benchmark whose score does discriminate needs no declaration.

What it found

Not hypotheticals. Both came out of pointing it at real work.

A model comparison that says nothing, and why that is worth knowing. Eighteen live agent runs, haiku-solo against sonnet-solo over sidekick-harness's three local tasks, three trials each:

$ evalplane compare leadmodel-haiku-solo leadmodel-sonnet-solo
  ✓  comparable: sidekick@1, 3 case(s), n=3

             haiku-solo   sonnet-solo   delta
mean score   1.000        1.000         +0.000
total cost   $0.30        $0.78         +0.48

  paired score delta  +0.000   CI [+0.000, +0.000]   within noise

Every case, every trial, both models: 1.0. The prediction was registered before the run and it held. The result is about the evalset rather than the models: those three tasks are saturated at the cheapest tier, so they cannot discriminate between models at all, and sonnet costs 2.6 times more to produce identical output. Useful to know before anyone reaches for them to compare models.

A caveat rediscovered from the data alone. sidekick-harness reports a striking pair of numbers, fable-solo at 1.00 against opus-solo at 0.50. Asking the platform to compare those two configs:

$ evalplane compare opus-solo@cc fable-solo@cc
  ✗  not comparable
    - case sets differ: 6 only in B

fable-solo ran seven tasks; opus-solo ran one. They overlap on a single case. That repo already says so, prominently and before any table: every cell is n=1, and it goes further, noting that fable itself scored both 1.00 and 0.50 on that same task across runs, so the run-to-run variance is as large as the measured difference between models.

The point is not that the caveat was missing. It is that the guard derived it from the run records alone, with no access to that prose. A hand-written footnote in one repo became a structural check that fires in another.

How a benchmark is defined

Three functions, because the three repos differ in all three and share nothing else. Only the queue, the run record and the comparison are actually common.

from evalplane import benchmark, runner, grader, Case, RunOutput, Score

@benchmark("review-lab", version=2)
def cases():
    return [Case(id="clean-briefing"), Case(id="hard-cache-stale")]

@runner("review-lab")
def run(case, config) -> RunOutput:
    ...  # drive the agent however this benchmark drives agents
    return RunOutput(workspace=ws, artifacts={"transcript": path},
                     cost_usd=1.25, metrics={"turns": 14})

@grader("review-lab")
def grade(out: RunOutput) -> Score:
    return Score(value=0.8, checkpoints={"tests": 1.0})

Existing graders run unmodified through an escape hatch, which is how rl-environments' 22 grade.py files were adopted without rewriting any of them:

grader("partial-refund-cent-loss",
       grade_cmd="python3 grade.py --solution {ws} --out {out}")

Architecture

  Go server                        Python worker(s)
  ---------                        ----------------
  POST /experiments   -> enqueue cases x trials
                              <-  POST /runs/claim        (lease)
                                    @runner(case, config)
                                    @grader(output)
                              <-  POST /runs/{id}/heartbeat
                              <-  PUT  /artifacts/{sha256}
                              <-  POST /runs/{id}/complete
  lease expiry -> requeue, or quarantine past the retry budget

• The queue is Postgres, not a broker. SELECT … FOR UPDATE SKIP LOCKED, with leases, heartbeats, at-least-once delivery, a retry budget and poison quarantine. The workload is thousands of multi-minute runs; Kafka would be an operational dependency bought for nothing.
• Workers run anywhere with the server URL and a token. The platform never learns how an agent works, because that is the benchmark's business.
• Artifacts are content-addressed. A repeated sweep writes the same transcripts over and over; 9 demo runs deduplicate to 3 blobs on disk.
• Benchmark content stays in its own repo, in git. evalplane stores a reference (review-lab@2), never a copy.
• Aggregates are computed on read, never stored, so results re-slice by model, category or cost without re-running anything.

Running it

createdb evalplane_dev
make build
EVALPLANE_DSN=postgres://localhost:5432/evalplane_dev ./bin/evalplane-server

The dashboard is at /, the API alongside it. make test needs evalplane_test to exist.

evalplane experiments ls
evalplane runs ls --benchmark review-lab --since 7d
evalplane compare I1-medium I1-high

Workers load a benchmark module and pull work:

evalplane-worker --path examples --import demo_benchmark --worker-id laptop-1

examples/demo_benchmark.py is a self-contained benchmark with a deterministic stub agent, so the whole loop runs with no API key.

Deploying

Set EVALPLANE_TOKEN. The server refuses to bind a non-loopback address without one. Forgetting is easy, and noticing an open eval platform on the internet is not.

Locally, docker compose up brings up Postgres and the server together, which is how this is actually run. There is no hosted instance: the dashboard is a tool for whoever holds the data, so the screenshots above are the demo and a public URL would only add a bill.

scripts/deploy-fly.sh [app] [region] is there if that changes. It provisions the app, a Postgres machine, the artifact volume and a generated token, then deploys, and every step checks whether the resource already exists so a re-run after a partial failure continues rather than duplicating anything. Run flyctl auth login first; that step opens a browser and cannot be scripted.

Variable	Default	Purpose
EVALPLANE_DSN	none	Postgres connection string (required)
EVALPLANE_ADDR	127.0.0.1:8080	listen address
EVALPLANE_TOKEN	none	bearer token, required off loopback
EVALPLANE_BLOB_ROOT	./blobstore	artifact directory
EVALPLANE_SWEEP_INTERVAL	30s	lease reclaim interval
EVALPLANE_MAX_ATTEMPTS	3	retries before quarantine

What it does not do

Stated here rather than discovered later:

• It cannot make an experiment clean. It records that a benchmark changed. It cannot tell you a treatment was confounded, and no infrastructure can.
• Delivery is at-least-once, so a non-idempotent grader can double-count.
• n is usually small. The statistics make that visible; they do not fix it.
• It does not own a container runtime. Workers execute in whatever environment the task declares. Sandboxing is a separate problem.
• Single user. One shared token, no roles.
• Three repos is a small federation. The abstraction is validated against three real consumers, which is enough to be honest and not enough to claim generality.

Imported data

Import is read-only: the source repos are never modified, their verify scripts keep working, and their published numbers do not move. Every imported row carries source='import:<repo>', the commit it was read at in source_ref, and its original path and timestamp. The commit is deliberately kept out of the identity: import idempotency keys on the source, so folding the commit in meant any change to the source repo produced a second copy of the corpus instead of replacing it.

Values the sources did not record are imported as missing, never invented. review-agent-lab records tokens but not dollars, so its cost is null rather than back-computed from a price table it never used. Contaminated and aborted rl-environments runs import with a null score, because that repo explicitly voids them. Four runs predating task_version import as v0, so the guard refuses to compare them with current results.

For why it is shaped this way, and which parts of it are ordinary, see docs/architecture.md.

Layout

server/   Go: API, Postgres store, queue, sweeper, blob store, dashboard
sdk/      Python: benchmark registration, worker, HTTP client
cli/      Go: evalplane runs ls, experiments ls, compare
importers/    one per source repo, read-only
integrations/ adapters that run an existing harness through the platform
examples/     a self-contained benchmark needing no API key
docs/plans/   design doc and the weekend-1 implementation plan

Prior art worth naming: the dataset/solver/scorer split matches UK AISI's inspect. The comparability guard and the statistics layer are the parts I have not seen elsewhere, and they came from being burned. An earlier writeup of mine carries a hand-written footnote that a 100% to 60% drop was rig-dependent. This makes that footnote structural.