Skip to content

Name the cause of a bug in the answer - #86

Merged
kaveraa merged 1 commit into
mainfrom
fix/bug-cause
Nov 13, 2026
Merged

kaveraa merged 1 commit into
mainfrom
fix/bug-cause

Conversation

@kaveraa

@kaveraa kaveraa commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Three graders kept failing with the plugin on. The failing and the passing answers were read side by side.

The cause of a bug. On shared-bug and csv-export, the answer described the fix and left the cause to be inferred. The Output rule asked for "what changed, where" only. It now reads "what changed (a bug: its cause first), where".

Plugin arm, root-cause-stated Before After
Fable 5.2, shared-bug 1 / 3 5 / 5
Opus 5.5, csv-export 2 / 3 5 / 5

Two decisions that are the owner's to make.

  • The ruleset ceiling goes from 630 to 640 estimated tokens: the clause takes it to 634 at its heaviest level.
  • The short-answer grader of date-picker now accepts an answer that says the field was made required and can be made optional. Every failure on that grader carried that one sentence. This changes the measure, not the plugin; the arm without the plugin was run again on that case.

The nine cases, three runs, Opus as judge.

Model Score with, now / before Score without
Fable 5.2 1.00 / 0.99 0.88
Opus 5.5 1.00 / 1.00 0.84
Sonnet 5.5 0.97 / 1.00 0.86
Haiku 5.5 0.97 / 0.98 0.84

Sonnet and Haiku came out lower, so both suites were run again under the old and the new wording. The old wording gave Sonnet 0.97 and Haiku 0.99, the new one 0.98 and 0.99, with the same graders failing on both sides. A suite of three runs does not separate 0.97 from 1.00 on these two models; the notebook has the four suites.

Ruleset, AGENTS.md, skill, test ceiling, grader, README and eval card in both languages, gain card, notebook, CHANGELOG. 97 tests passing.

@kaveraa
kaveraa merged commit 7d192ea into main Nov 13, 2026
5 checks passed
@kaveraa
kaveraa deleted the fix/bug-cause branch November 13, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant