Repository navigation
Name the cause of a bug in the answer - #86
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three graders kept failing with the plugin on. The failing and the passing answers were read side by side.
The cause of a bug. On
shared-bugandcsv-export, the answer described the fix and left the cause to be inferred. The Output rule asked for "what changed, where" only. It now reads "what changed (a bug: its cause first), where".root-cause-statedshared-bugcsv-exportTwo decisions that are the owner's to make.
short-answergrader ofdate-pickernow accepts an answer that says the field was made required and can be made optional. Every failure on that grader carried that one sentence. This changes the measure, not the plugin; the arm without the plugin was run again on that case.The nine cases, three runs, Opus as judge.
Sonnet and Haiku came out lower, so both suites were run again under the old and the new wording. The old wording gave Sonnet 0.97 and Haiku 0.99, the new one 0.98 and 0.99, with the same graders failing on both sides. A suite of three runs does not separate 0.97 from 1.00 on these two models; the notebook has the four suites.
Ruleset, AGENTS.md, skill, test ceiling, grader, README and eval card in both languages, gain card, notebook, CHANGELOG. 97 tests passing.