Skip to content

Add an eval case for long text and comments - #82

Merged
kaveraa merged 1 commit into
mainfrom
evals/long-text
Nov 13, 2026
Merged

kaveraa merged 1 commit into
mainfrom
evals/long-text

Conversation

@kaveraa

@kaveraa kaveraa commented Nov 13, 2026

Copy link
Copy Markdown
Owner

The plain text and comment graders passed almost everywhere on the older cases, in both arms: short tasks do not invite long prose. explain-retry does: explain a fetch with retries step by step to a new hire, and add the comments the code needs. Fourteen graders, twelve of them regular expressions.

Runs passed, with / without the plugin Fable Opus Sonnet Haiku
full-explanation 3 / 3 3 / 3 3 / 3 3 / 3
plain-text-answer 3 / 0 3 / 0 1 / 0 3 / 0
comments-one-line 3 / 0 3 / 0 3 / 0 2 / 0
comments-say-why 3 / 2 1 / 0 1 / 0 0 / 0

Without the plugin no model writes a plain answer or one-line comments. With it the explanation stays complete in every run. Two weak spots show: Sonnet keeps a long dash in 2 answers out of 3, and comments-say-why passes in 5 runs out of 12, because one comment per file often names the mechanism instead of the reason. The notebook quotes the comments.

The eval tables and the eval card count nine cases, joined from measures of the same day on the same plugin code, as for the eight-case table. README and cards in both languages, gain card, notebook, CHANGELOG. 97 tests passing.

@kaveraa
kaveraa merged commit a7fd2c9 into main Nov 13, 2026
5 checks passed
@kaveraa
kaveraa deleted the evals/long-text branch November 13, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant