Skip to content

Add an eval case for current practice - #78

Merged
kaveraa merged 1 commit into
mainfrom
evals/current-practice
Nov 13, 2026
Merged

kaveraa merged 1 commit into
mainfrom
evals/current-practice

Conversation

@kaveraa

@kaveraa kaveraa commented Nov 13, 2026

Copy link
Copy Markdown
Owner

The rule "follow the installed versions' current docs" had no case. zod-email gives one: the manifest declares zod 4, the ticket asks for an email field, and nothing in the file shows that zod 4 replaced z.string().email() with z.email(). Ten graders, all regular expressions.

Model Current API with / without Score with / without
Fable 5.2 3 of 3 / 3 of 3 1.00 / 1.00
Opus 5.5 3 of 3 / 3 of 3 1.00 / 1.00
Sonnet 5.5 3 of 3 / 0 of 3 1.00 / 0.80
Haiku 5.5 3 of 3 / 0 of 3 1.00 / 0.80

With the versions line taken out of a local copy of the hook, the plugin arm gets 2 of 3 on Sonnet and on Haiku: the rule does most of it, the line added in 0.14.0 closes the gap. Three runs per cell.

A note for later measures is in the notebook: CREWCUT_MAP=off does not reach the hook during claude plugin eval.

The published table stays over the seven older cases. 97 tests passing.

@kaveraa
kaveraa merged commit 3a9a579 into main Nov 13, 2026
5 checks passed
@kaveraa
kaveraa deleted the evals/current-practice branch November 13, 2026 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant