An Agent Skill for designing comprehensive AI evaluation strategies using Evaluation-Driven Development (EDD).
Created by Bayram Annakov while building Onsa.ai - AI agents for B2B sales prospecting.
If you find this useful, say hi on LinkedIn!
Eval Coach guides you through a structured 5-step framework for evaluating LLM applications:
- Define Success - Map business goals to measurable metrics
- Design Dataset - Create diverse test cases (happy path, edge cases, adversarial)
- Select Methods - Choose Automated, LLM-as-Judge, or Human evaluation
- Plan Automation - Integrate evals into CI/CD
- Monitor Production - Track drift and collect feedback
Copy this skill to your project's skills directory:
git clone https://github.com/BayramAnnakov/eval-coach.git ~/.claude/skills/eval-coachOr add to your project:
mkdir -p skills
git clone https://github.com/BayramAnnakov/eval-coach.git skills/eval-coachInvoke the skill by name or with trigger keywords:
/eval-coach
Or just mention evaluation-related topics:
- "help me create an evaluation strategy"
- "design test cases for my agent"
- "set up LLM testing"
A practical distribution for evaluation methods:
| Tier | Method | Cost | Percentage |
|---|---|---|---|
| 1 | Automated (schema, keywords, latency) | $0.00/run | 50% |
| 2 | LLM-as-Judge (quality, relevance) | $0.01-0.05/run | 40% |
| 3 | Human Review | $5-50/run | 10% |
templates/dataset.py- LangSmith dataset creation with example test casestemplates/evaluators.py- 10 ready-to-use evaluators (automated, LLM-as-Judge, performance)templates/compare.py- Experiment comparison utilities
The most dangerous failures are the ones your tests don't catch. The input_data_consistency_evaluator was born from a real production issue:
We ran an agent with
--target john_smith --company CompanyA, but John actually works at CompanyB. The agent silently reconciled the contradiction by writing "engaged with CompanyA via partnerships" instead of flagging the mismatch. All 9 evaluators passed, but the output was misleading.
Always include adversarial test cases that probe for silent failures.
This skill follows the Agent Skills open standard (SKILL.md format), supported by:
- Claude Code
- Cursor
- VS Code with AI extensions
- Gemini CLI
- OpenAI Codex
For the templates:
- Python 3.9+
- LangSmith account (for dataset management)
- Google API key (for LLM-as-Judge evaluators)
pip install langsmith langchain-google-genaiSet the judge model via environment variable:
export GOOGLE_API_KEY="your-api-key"
export JUDGE_MODEL="gemini-3-flash-preview" # Default (fast, cost-effective)
export JUDGE_MODEL="gemini-3-pro-preview" # For higher quality judgingMIT License - see LICENSE