Add build-a-judge skill for designing MLflow scorer suites - #35
Open
joshuawong-db wants to merge 3 commits into
Open
Add build-a-judge skill for designing MLflow scorer suites#35joshuawong-db wants to merge 3 commits into
joshuawong-db wants to merge 3 commits into
Conversation
Judge design is the hardest and most opinionated part of evaluation, so it gets its own composable skill rather than another section inside agent-evaluation. The skill encodes a point of view on how to build judges, drawn from evaluation practice rather than left to the model's defaults: - Small suites beat checklists. 3-5 scorers, hard cap; adding one means dropping a weaker one, because unvalidated checks are worse than none. - One scorer, one criterion. Compound "overall quality" judges cannot be debugged when they fail. - Cheapest reliable implementation wins. Code/rules beat built-ins beat hand-written LLM judges; every judge must name the code check and the built-in it rejected, so LLM-judge-by-default is not an option. - Measure outcomes, not only policy. An agent can pass every compliance check while failing everyone who used it. - Binary outputs. Only bool, numerics, and "yes"/"no" survive aggregation. - Treat the first LLM judge as a draft, to be aligned once traces and human labels exist. - Talk before code. A wall of implementation ends the design conversation, so it comes last, after the criteria are confirmed. Composition, not duplication: agent-evaluation Step 2 defers here when the user does not yet know what to measure, and this skill hands back for scorer registration, dataset prep, execution, and analysis. Tracing gaps surfaced during design route to instrumenting-with-mlflow-tracing. Output-type guidance follows agent-evaluation/references/scorers-constraints.md, verified against mlflow/genai/scorers/aggregation.py: "pass", "fail", and "not_applicable" are silently cast to None and dropped from results.metrics. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Joshua Wong <joshua.wong@databricks.com>
The two skills overlapped in four places: interviewing the user, deciding criteria, choosing an implementation per criterion, and validating the suite on a small sample. Overlapping instructions mean whichever skill loads first wins, so the split is now explicit. build-a-judge owns WHAT to measure: eliciting and sharpening criteria, atomicity, rule/outcome coverage, routing each criterion to a code check, built-in, or LLM judge, and emitting the implementation. Its final phase now names the validation plan and hands off rather than telling the user to run an evaluation, which duplicated agent-evaluation Step 3.5. agent-evaluation owns HOW to run it: registration, dataset discovery and creation, dry run, execution, and analysis. Step 1 no longer derives scorer names from three interview questions — those answers establish what the agent is, not what good looks like — and Step 2 is renamed to "Register Quality Scorers" to match what it actually does. The seam is a confirmed suite of criteria with an implementation chosen for each. Both skills state the boundary explicitly so an agent that drifts across it knows to hand off. Also corrects the built-ins guidance, which over-claimed. Built-ins are LLM judges carrying MLflow's generic instructions, so "maintained upstream" does not imply "correct for your product". Verified against mlflow/genai/judges/base.py: align() is defined on Judge, so single-turn built-ins can be aligned to user labels like make_judge can, while session-level built-ins raise NotImplementedError and are permanently stuck on MLflow's definition. Guidelines remains the exception that always applies because it takes the user's own policy text. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Joshua Wong <joshua.wong@databricks.com>
There was a problem hiding this comment.
🟡 Changes recommended
The new skill’s scorer-introspection example conflicts with its own uv run guidance, which could lead to incorrect copy/paste usage.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
Adds a new build-a-judge skill to guide users through designing a small, debuggable MLflow scorer/judge suite, and wires it into the existing agent-evaluation workflow and skill registry.
Changes:
- Introduces
build-a-judge/SKILL.mdwith an opinionated, phased process for eliciting criteria and routing each to code checks, built-ins, or LLM judges. - Updates
agent-evaluation/SKILL.mdto explicitly hand off scorer-suite design tobuild-a-judgeand reframe Step 2 around registering an agreed suite. - Registers the new skill in
.claude-plugin/plugin.jsonand lists it inREADME.md.
File summaries
| File | Description |
|---|---|
| README.md | Adds build-a-judge to the skills list so users can discover it. |
| build-a-judge/SKILL.md | New skill content defining doctrine, workflow phases, and routing rules for MLflow scorers/judges. |
| agent-evaluation/SKILL.md | Clarifies scope boundary and adds explicit handoff to build-a-judge when criteria aren’t yet defined. |
| .claude-plugin/plugin.json | Registers ./build-a-judge so the plugin loads the new skill. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 1
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
The prose said to use `uv run python` while the example used bare `python -c`, so a copy/paste would contradict the instruction. The repo documents `uv run python -c "..."` for one-liners in agent-evaluation's Command Conventions, and uses it in 13 places against 2 bare invocations. Verified the command runs as written. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Joshua Wong <joshua.wong@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
build-a-judge, an opinionated skill for users to use when building a judge.There is some overlap with
agent-evaluationon the building judges part, but this is a specialized and standalone skill.build-a-judgeagent-evaluationI've broken apart both skills and got
agent-evaluationto referencebuild-a-judgeinstead.Why a separate skill
Starting from zero is really hard, and there are nuances about evaluation that people new to agentic development are not familiar with. We should guide new users along a golden path as much as possible.
Also in general, agents stop following instructions once a skill gets too long, so a standalone skill might help with adherence to guidelines.
I have listed down some opinions, of which some are new.
I want judge creation to be a conversation between user and LLM. The Skill needs to distill the user notion of right and wrong into a small set of judges which they can iterate on. It doesn't have to be perfect, just good enough to start work on.
Testing
Testing was done manually. I passed in a starting scenario into an agent, and had the agent invoke this skill and store the transcript. I then inspected the transcript to see if I agreed with the scorer suite and iterated until a satisfactory outcome was reached.
Future Extensions
For future extensions, I think we can consider outsourcing this flow into the UI, or via Assistant for a lower lift approach and for better interaction with the eval user flow. But this is comparatively less important.
🤖 Generated with Claude Code