Skip to content

Publish the eval tables on six runs - #88

Merged
kaveraa merged 1 commit into
mainfrom
evals/six-runs
Nov 13, 2026
Merged

kaveraa merged 1 commit into
mainfrom
evals/six-runs

Conversation

@kaveraa

@kaveraa kaveraa commented Nov 13, 2026

Copy link
Copy Markdown
Owner

A suite of three runs did not separate 0.97 from 1.00 on Sonnet and Haiku. A second full suite was run, both arms, four models, and every run measured with the 0.17.0 wording and graders is pooled: six runs per case, nine with the plugin on Sonnet and Haiku.

Model Score with without Runs with without
Fable 5.2 1.00 0.87 54 54
Opus 5.5 0.99 0.85 54 54
Sonnet 5.5 0.98 0.87 81 54
Haiku 5.5 0.99 0.84 81 54

No change to the plugin. README and eval card in both languages, gain card, notebook with every grader that still fails with the plugin, CHANGELOG. 97 tests passing.

@kaveraa
kaveraa merged commit 5230ae4 into main Nov 13, 2026
5 checks passed
@kaveraa
kaveraa deleted the evals/six-runs branch November 13, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant