diff --git a/README.md b/README.md index 88d649a..0ca3ea2 100644 --- a/README.md +++ b/README.md @@ -165,6 +165,7 @@ Most "awesome" lists are link dumps. This one is **annotated and verified**: eve - **[UpTrain](https://github.com/uptrain-ai/uptrain)** — — 20+ preconfigured checks + root-cause analysis on failures. - **[HF `evaluate`](https://github.com/huggingface/evaluate)** — — classic metrics library, ⚠️ maintenance mode (use lighteval for LLMs). - **[Harbor](https://github.com/harbor-framework/harbor)** — harbor-framework (Laude Institute / Stanford) — — 🆕 framework for running agent evals + creating/using RL environments; powers Terminal-Bench 2.0. ~2.7k★. ⚠️ name overloaded (cf. `av/harbor` local-LLM toolkit). +- **[Coder Eval](https://github.com/UiPath/coder_eval)** — UiPath — · — 🆕 runs a real coding agent (Claude Code / Codex / Gemini) in a sandbox against declarative YAML tasks and grades the artifacts *and* the trajectory it produced: 14 criterion types, weighted 0.0–1.0 credit, per-tool token/USD telemetry, and a 5-layer A/B variant resolver so model-vs-model or skill-on-vs-off runs share one task set. Its distinguishing primitive is treating **skill activation as a measurable variable** — a `skill_triggered` criterion fanned over a labelled dataset yields suite-level precision/recall/F1 thresholds, catching the silent retrieval regressions that outcome-only scoring charges to the model (cf. §3). Harness-level, not leaderboard-level: the suite is your own tasks and skills. ~0.1k★. ### 5b · TypeScript/JS-native eval runners - **[evalite](https://github.com/mattpocock/evalite)** — Matt Pocock — — 🆕 local-first eval runner on Vitest; `.eval.ts` files, web UI, cost-aware.