Add Coder Eval to 5a · Eval frameworks & harnesses

[uipreliga]

Adds Coder Eval to §5a · Eval frameworks & harnesses (code-first test-runners).

What it is

An Apache-2.0 harness that runs a real coding agent (Claude Code, Codex CLI, Gemini/Antigravity) in a sandbox against declarative YAML tasks, then grades both the artifacts and the trajectory it produced. pip install coder-eval, currently 0.8.x.

Why it clears the bar

It is a framework, not a take. Code-first and git-diffable in the same sense as promptfoo: the unit of work is a YAML task with typed success criteria, and the runner is a CLI you put in CI. 14 criterion types spanning deterministic checks (file/JSON assertions, regex, exit codes, executed-command verification), continuous scoring (AST/token/complexity similarity against a reference), and graded rubrics (llm_judge, plus an agent_judge that spawns a tool-using sub-agent in an isolated sandbox copy). Weighted 0.0–1.0 credit with per-criterion pass thresholds rather than binary pass/fail.

It fills a gap in the section. §5a's existing entries evaluate a model through a harness the framework itself provides. Coder Eval inverts the system-under-test: the agent harness (Claude Code and its skills, its tools, its prompt) is what varies, and the task suite is held fixed. A 5-layer merge resolver (default → experiment defaults → task → variant → CLI) makes that concrete — you declare variants and get model-vs-model, skill-on-vs-skill-off, or agent-vs-agent runs over one identical task set, with per-tool token and USD telemetry attached to every run.

It operationalizes §3 directly. This section-3 already argues same model, different harness, opposite ranking (Brand's AlgoTune case, Lee's "harness is the agent", HAL's standardized-harness answer). Coder Eval's distinguishing primitive is measuring one specific harness variable that those readings identify but no listed tool instruments: skill retrieval. A skill_triggered criterion fanned over a labelled dataset emits per-row classification results, and suite_thresholds gates the suite on accuracy/precision/recall/F1. The check is agent-agnostic — Claude's Skill tool call, or on Codex the agent reading the skill's files off disk — so it survives the cross-vendor tool-schema problem rather than measuring one vendor's telemetry shape.

That matters because skill-retrieval failure is invisible to outcome-only scoring: the symptom is a lower task score, which teams charge to the model. Making activation a first-class labelled classification problem turns it into a CI gate instead.

Honest caveats

• Small and young (~0.1k★, first release 2026). Listed with ~0.1k★ in the entry so readers can size it themselves — happy to drop the entry if you'd rather wait for adoption.
• Python 3.13+ only.
• Scoped to coding agents specifically — it is not a general-purpose LLM eval runner, which is why §5a rather than §5f.

Links

• Repo: https://github.com/UiPath/coder_eval
• Docs: https://coder-eval.com/docs · criterion reference · A/B experiments · CI gate
• PyPI: https://pypi.org/project/coder-eval/

Format follows the section's existing convention (- **[Name](url)** — Org — <url> · <docs> — 🆕 note.). One change, one PR, README only.

If a deep note under notes/ would be useful, we have unpublished measurements that fit §3 and §6 — most usefully a case where the first cross-model run of a suite scored 44% vs 85%, and a hand-audit of all 227 failures attributed ~89% of the gap to harness/environment causes (a broken file-edit tool, a different sandbox OS, expired credentials, and an activation check wired to one vendor's tool schema) rather than to model capability. Happy to write that up separately if you want it.

Disclosure: I'm a maintainer of the project. Happy to reword, shorten, move sections, or withdraw.

[xdotli]

This is the strongest tool submission in the current queue and I want it in. Verified independently: UiPath is a verified GitHub org, UiPath/coder_eval is at 107★ with a push today, Apache-2.0 with NOTICE and SECURITY.md, and both URLs resolve clean. Your maintainer disclosure checks out too — you're the #1 contributor. The "system under test is the harness, not the model" framing fills a real gap against §3.

Two changes before merge:

1. Length. The entry runs about 3× its neighbors in §5a. Please trim to a single line in the CONTRIBUTING format. The skill-activation-as-a-measurable-variable idea is the distinguishing hook — lead with that and cut the rest.

2. Unpublished claims. The entry asserts precision/recall/F1 thresholds for skill retrieval, but the docs confirm there's no canonical scored dataset ("Coder Eval scores your tasks"). Please soften anything that isn't publicly checkable.

Separately: the 44% vs 85% cross-model case with 227 hand-audited failures that you mentioned would make an excellent deep note under notes/ — that's the kind of war story §3 and §6 are short on. Entirely optional, and not a condition of merging.

[xdotli]

Correction — I quoted something you never wrote.

In my item 2 I put "Coder Eval scores your tasks" in quotation marks as if it came from your docs. It doesn't — I paraphrased and then quoted the paraphrase. I checked: that string appears nowhere in coder-eval.com/docs. On a list whose whole premise is that quotes are verbatim, that's exactly the error I've been closing other PRs for, so I'm sorry.

Withdrawing item 2 entirely. Re-reading the entry, it doesn't actually assert unpublished numbers — it describes a documented capability and already ends with "Harness-level, not leaderboard-level: the suite is your own tasks and skills," which is the softening I was asking for. It was already there.

Item 1 stands unchanged: the entry is about 3× the length of its §5a neighbours. Trim it to one line and I'll merge.