As we attempt to generate reliable data to prove the effectiveness of our skills, we've managed to learn a lot about writing good evals, but we continue to struggle to actually generate the desired data - this has led to a large change in the eval-magic dependency to support more realistic environments to run evals in, as well as a change in how we think about the goals and purposes of running skill evals.
I think we should consider a overall rewrite of evaluating-skills, moving past things like Mode A vs Mode B evals and ablation/decomposition, with a clearer focus on what evals are trying to check for, when to run them, and how to author an eval suite that provides regression testing and performance improvement benchmarks.
Value props and coverage
I think it's worth clarifying what kind of value props evals are intended to provide, and what the goal of our coverage report is. Every skill should have a motivating force - the problem that introducing the skill should fix, or the process it should improve. Evals are designed to determine if the skill is satisfying the requirements of the motivating force.
I think this changes how we should consider eval coverage - we're not trying to cover all of the skill's written behavior, but all of its intended goals. While these intended goals might not be completely fixed, they should be relatively unchanging. This means that reaching complete coverage should be a reasonable goal for any skill, which should be focused anyway.
This also means that completely new eval cases should be relatively rare, with the complete coverage suite functioning as both regression tests to avoid changes worsening behavior, as well as the benchmarks by which improvement can be measured against. When changes are made to a skill, running the existing suite should be enough to show the results of the change, unless the change introduces completely new goals for the skill. This should mean that there is no concept of Mode A vs Mode B evals (which is still relevant for eval-magic to know how to run the evals), and eval authoring becomes a part of the skill authoring process, and shouldn't need to be constantly iterated on.
Here's a concrete example. Agents often start working on the branch they're currently on, without looking at existing changes - slow-powers:working-in-isolation has the goal of making sure an agent works in the correct kind of isolation. An eval suite that checks that a workspace is used under the right circumstances, work isn't done on the default branch without permission, and so on, should count as complete coverage. I could rewrite the prose of the skill from scratch, and the eval suite can tell me if the rewrite had any effect, without changing the evals themselves. If working-in-isolation was updated to also handle deleting any branches that were created when following its instructions, this would be a new goal, requiring new evals to achieve complete coverage.
New eval process
Ultimately, I think this guidance supports a new process for authoring evals that's simpler and more straightforward than most of what our skill currently suggests. The goal should be to come up with a suite of evals that covers all the things the skill is trying to achieve, iterate through the suite by doing test runs until it can be shown to be a reliable baseline of data, then providing guidance on the rare changes where new eval cases are warranted.
Evals can, of course, have issues, and need further iteration in the future. "Complete coverage" is the guiding light we're following, not an expected end state. But the evals written and the data generated during the authoring of the suite should be able to convince someone that the skill does it's job, and improves behavior in some way.
CONTINUE HERE
Grading metrics
This anthropic article has some interesting content in general, but is overall more focused on testing agents than skills. However, the section "How to think about non-determinism in evaluations for agents" has some interesting content for us, along with two linked research articles.
https://proceedings.neurips.cc/paper/2019/file/7298332f04ac004a0ca44cc69ecf6f6b-Paper.pdf
https://arxiv.org/pdf/2406.12045
While both short, I have to admit that reading these papers is a bit beyond me (I tried, but I was never good at research). However, based on the summary in the anthropic article, I think it may be valuable to synthesize some of this content, and see if there's any value to the pass@k vs pass^k grading split, that we can include in our grading guidance. eval-magic should now support the ability to run pass^k grading, which should be the main technical gap.
As we attempt to generate reliable data to prove the effectiveness of our skills, we've managed to learn a lot about writing good evals, but we continue to struggle to actually generate the desired data - this has led to a large change in the eval-magic dependency to support more realistic environments to run evals in, as well as a change in how we think about the goals and purposes of running skill evals.
I think we should consider a overall rewrite of
evaluating-skills, moving past things like Mode A vs Mode B evals and ablation/decomposition, with a clearer focus on what evals are trying to check for, when to run them, and how to author an eval suite that provides regression testing and performance improvement benchmarks.Value props and coverage
I think it's worth clarifying what kind of value props evals are intended to provide, and what the goal of our coverage report is. Every skill should have a motivating force - the problem that introducing the skill should fix, or the process it should improve. Evals are designed to determine if the skill is satisfying the requirements of the motivating force.
I think this changes how we should consider eval coverage - we're not trying to cover all of the skill's written behavior, but all of its intended goals. While these intended goals might not be completely fixed, they should be relatively unchanging. This means that reaching complete coverage should be a reasonable goal for any skill, which should be focused anyway.
This also means that completely new eval cases should be relatively rare, with the complete coverage suite functioning as both regression tests to avoid changes worsening behavior, as well as the benchmarks by which improvement can be measured against. When changes are made to a skill, running the existing suite should be enough to show the results of the change, unless the change introduces completely new goals for the skill. This should mean that there is no concept of Mode A vs Mode B evals (which is still relevant for eval-magic to know how to run the evals), and eval authoring becomes a part of the skill authoring process, and shouldn't need to be constantly iterated on.
Here's a concrete example. Agents often start working on the branch they're currently on, without looking at existing changes -
slow-powers:working-in-isolationhas the goal of making sure an agent works in the correct kind of isolation. An eval suite that checks that a workspace is used under the right circumstances, work isn't done on the default branch without permission, and so on, should count as complete coverage. I could rewrite the prose of the skill from scratch, and the eval suite can tell me if the rewrite had any effect, without changing the evals themselves. Ifworking-in-isolationwas updated to also handle deleting any branches that were created when following its instructions, this would be a new goal, requiring new evals to achieve complete coverage.New eval process
Ultimately, I think this guidance supports a new process for authoring evals that's simpler and more straightforward than most of what our skill currently suggests. The goal should be to come up with a suite of evals that covers all the things the skill is trying to achieve, iterate through the suite by doing test runs until it can be shown to be a reliable baseline of data, then providing guidance on the rare changes where new eval cases are warranted.
Evals can, of course, have issues, and need further iteration in the future. "Complete coverage" is the guiding light we're following, not an expected end state. But the evals written and the data generated during the authoring of the suite should be able to convince someone that the skill does it's job, and improves behavior in some way.
CONTINUE HERE
Grading metrics
This anthropic article has some interesting content in general, but is overall more focused on testing agents than skills. However, the section "How to think about non-determinism in evaluations for agents" has some interesting content for us, along with two linked research articles.
https://proceedings.neurips.cc/paper/2019/file/7298332f04ac004a0ca44cc69ecf6f6b-Paper.pdf
https://arxiv.org/pdf/2406.12045
While both short, I have to admit that reading these papers is a bit beyond me (I tried, but I was never good at research). However, based on the summary in the anthropic article, I think it may be valuable to synthesize some of this content, and see if there's any value to the
pass@kvspass^kgrading split, that we can include in our grading guidance.eval-magicshould now support the ability to runpass^kgrading, which should be the main technical gap.