diff --git a/CHANGELOG.md b/CHANGELOG.md index 1f1b2e8..a4336ee 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,22 @@ follows Keep a Changelog and the project follows Semantic Versioning. ## [Unreleased] +### Added + +- A ninth eval case, `explain-retry`: a long explanation and comments to + write, where a model reaches for long dashes, arrows and comment blocks. + Without the plugin no model writes a plain answer or one-line comments, + 0 run out of 12 each. With it, 10 and 11 out of 12, and the explanation + stays complete in every run. Two weak spots: Sonnet 5.5 keeps a long + dash in 2 answers out of 3, and one comment per file often names the + mechanism instead of the reason. + +### Changed + +- The eval tables and the eval card count nine cases: Fable 5.2 0.98 + against 0.88 without the plugin, Opus 5.5 0.99 against 0.85, Sonnet 5.5 + 0.98 against 0.86, Haiku 5.5 0.97 against 0.84. + ## [0.15.7] - 2026-11-13 ### Changed diff --git a/README.fr.md b/README.fr.md index 559e560..a782191 100644 --- a/README.fr.md +++ b/README.fr.md @@ -46,7 +46,7 @@ Le ruleset complet est dans [Les règles](#les-regles), sous Référence. ## Mesuré -

Suite d'évals, huit cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,98 avec crewcut, 0,89 pour le même modèle sans plugin. Opus 5.5 : 0,99 avec, 0,86 sans. Sonnet 5.5 : 0,99 avec, 0,87 sans. Haiku 5.5 : 0,98 avec, 0,86 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro.

+

Suite d'évals, neuf cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,98 avec crewcut, 0,88 pour le même modèle sans plugin. Opus 5.5 : 0,99 avec, 0,85 sans. Sonnet 5.5 : 0,98 avec, 0,86 sans. Haiku 5.5 : 0,97 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro.

Le benchmark de ponytail 5, crewcut 0.14.0 face à sans plugin, Opus 5.5, 39 tâches, 5 runs. Lignes de code moins 58 pour cent, intervalle à 95 pour cent de moins 68 à moins 47, mieux sur 32 tâches et pire sur 1. Information de code moins 52 pour cent. Tokens de sortie moins 56 pour cent. Coût moins 34 pour cent, intervalle de moins 44 à moins 24. Temps moins 53 pour cent. Tours moins 19 pour cent. Vérifications cachées : 85 sur 90 réussies, 86 sans le plugin. Six tâches de sécurité : 30 sur 30 dans les deux bras. Tâche de confidentialité : 0 sur 5, puis 8 sur 10 après la règle qui nomme ce qu'un changement peut exposer. Ponytail 5 a publié pour lui-même : code moins 53 pour cent, tokens de sortie moins 45, coût moins 26, temps moins 41. Deux runs à un mois d'écart, les intervalles se recouvrent.

@@ -66,15 +66,15 @@ par tâche, limites et reproduction :
Les chiffres en texte -Suite d'évals, huit cas, trois runs chacun, Opus 5.5 comme juge. Le coût est +Suite d'évals, neuf cas, trois runs chacun, Opus 5.5 comme juge. Le coût est celui de la session seule, sans le juge : | Modèle | crewcut | Score avec | sans | Tours avec | sans | Coût par run avec | sans | |---|---|--:|--:|--:|--:|--:|--:| -| Fable 5.2 | 0.15.6 | 0,98 | 0,89 | 4,3 | 6,0 | 0,197 USD | 0,189 USD | -| Opus 5.5 | 0.15.6 | 0,99 | 0,86 | 4,8 | 5,7 | 0,089 USD | 0,086 USD | -| Sonnet 5.5 | 0.15.6 | 0,99 | 0,87 | 4,4 | 5,6 | 0,043 USD | 0,040 USD | -| Haiku 5.5 | 0.15.6 | 0,98 | 0,86 | 5,3 | 6,4 | 0,003 USD | 0,003 USD | +| Fable 5.2 | 0.15.7 | 0,98 | 0,88 | 4,3 | 5,7 | 0,203 USD | 0,192 USD | +| Opus 5.5 | 0.15.7 | 0,99 | 0,85 | 4,7 | 5,4 | 0,090 USD | 0,087 USD | +| Sonnet 5.5 | 0.15.7 | 0,98 | 0,86 | 4,4 | 5,3 | 0,044 USD | 0,041 USD | +| Haiku 5.5 | 0.15.7 | 0,97 | 0,84 | 5,2 | 6,0 | 0,003 USD | 0,003 USD | Le benchmark de ponytail 5, Opus 5.5, 39 tâches, 5 runs, crewcut 0.14.0 face à sans plugin : diff --git a/README.md b/README.md index b439dd8..7c12856 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ The full ruleset is in [The rules](#the-rules), under Reference. ## Measured -

Eval suite, eight cases, three runs each. Shorter, and more often right. Fable 5.2: 0.98 with crewcut, 0.89 for the same model with no plugin. Opus 5.5: 0.99 with, 0.86 without. Sonnet 5.5: 0.99 with, 0.87 without. Haiku 5.5: 0.98 with, 0.86 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero.

+

Eval suite, nine cases, three runs each. Shorter, and more often right. Fable 5.2: 0.98 with crewcut, 0.88 for the same model with no plugin. Opus 5.5: 0.99 with, 0.85 without. Sonnet 5.5: 0.98 with, 0.86 without. Haiku 5.5: 0.97 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero.

Ponytail 5's benchmark, crewcut 0.14.0 against no plugin, Opus 5.5, 39 tasks, 5 runs. Lines of code minus 58 percent, 95 percent interval minus 68 to minus 47, better on 32 tasks and worse on 1. Code information minus 52 percent. Output tokens minus 56 percent. Cost minus 34 percent, interval minus 44 to minus 24. Time minus 53 percent. Turns minus 19 percent. Hidden checks: 85 of 90 passed, 86 without the plugin. Six security tasks: 30 of 30 in both arms. Privacy task: 0 of 5, then 8 of 10 after the rule that names what a change could expose. Ponytail 5 published for itself: code minus 53 percent, output tokens minus 45, cost minus 26, time minus 41. Two runs a month apart, intervals overlap.

@@ -66,15 +66,15 @@ limits and how to reproduce:
The figures as text -Eval suite, eight cases, three runs each, Opus 5.5 as judge. The cost is the +Eval suite, nine cases, three runs each, Opus 5.5 as judge. The cost is the session alone, without the judge: | Model | crewcut | Score with | without | Turns with | without | Cost per run with | without | |---|---|--:|--:|--:|--:|--:|--:| -| Fable 5.2 | 0.15.6 | 0.98 | 0.89 | 4.3 | 6.0 | 0.197 USD | 0.189 USD | -| Opus 5.5 | 0.15.6 | 0.99 | 0.86 | 4.8 | 5.7 | 0.089 USD | 0.086 USD | -| Sonnet 5.5 | 0.15.6 | 0.99 | 0.87 | 4.4 | 5.6 | 0.043 USD | 0.040 USD | -| Haiku 5.5 | 0.15.6 | 0.98 | 0.86 | 5.3 | 6.4 | 0.003 USD | 0.003 USD | +| Fable 5.2 | 0.15.7 | 0.98 | 0.88 | 4.3 | 5.7 | 0.203 USD | 0.192 USD | +| Opus 5.5 | 0.15.7 | 0.99 | 0.85 | 4.7 | 5.4 | 0.090 USD | 0.087 USD | +| Sonnet 5.5 | 0.15.7 | 0.98 | 0.86 | 4.4 | 5.3 | 0.044 USD | 0.041 USD | +| Haiku 5.5 | 0.15.7 | 0.97 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD | Ponytail 5's benchmark, Opus 5.5, 39 tasks, 5 runs, crewcut 0.14.0 against no plugin: diff --git a/assets/readme/evals-fr.png b/assets/readme/evals-fr.png index b7591d4..a3de490 100644 Binary files a/assets/readme/evals-fr.png and b/assets/readme/evals-fr.png differ diff --git a/assets/readme/evals.png b/assets/readme/evals.png index 0ee9a84..a7ffcba 100644 Binary files a/assets/readme/evals.png and b/assets/readme/evals.png differ diff --git a/assets/readme/src/evals-fr.html b/assets/readme/src/evals-fr.html index 102822f..5310ecc 100644 --- a/assets/readme/src/evals-fr.html +++ b/assets/readme/src/evals-fr.html @@ -9,12 +9,12 @@ .legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
-
Suite d'évals, huit cas, trois runs chacun
+
Suite d'évals, neuf cas, trois runs chacun

Plus court, et plus souvent juste.

-
Fable 5.2
0,98
0,89
-
Opus 5.5
0,99
0,86
-
Sonnet 5.5
0,99
0,87
-
Haiku 5.5
0,98
0,86
+
Fable 5.2
0,98
0,88
+
Opus 5.5
0,99
0,85
+
Sonnet 5.5
0,98
0,86
+
Haiku 5.5
0,97
0,84
avec crewcutmême modèle, sans plugin
Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille :
une réponse plus courte mais fausse vaut zéro.
diff --git a/assets/readme/src/evals.html b/assets/readme/src/evals.html index cbd7990..dcb548e 100644 --- a/assets/readme/src/evals.html +++ b/assets/readme/src/evals.html @@ -9,12 +9,12 @@ .legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
-
Eval suite, eight cases, three runs each
+
Eval suite, nine cases, three runs each

Shorter, and more often right.

-
Fable 5.2
0.98
0.89
-
Opus 5.5
0.99
0.86
-
Sonnet 5.5
0.99
0.87
-
Haiku 5.5
0.98
0.86
+
Fable 5.2
0.98
0.88
+
Opus 5.5
0.99
0.85
+
Sonnet 5.5
0.98
0.86
+
Haiku 5.5
0.97
0.84
with crewcutsame model, no plugin
Score is the share of graders passed. Every case grades correctness as well as size:
a shorter answer that is wrong scores zero.
diff --git a/evals/RESULTS.md b/evals/RESULTS.md index 6a0eb12..512457e 100644 --- a/evals/RESULTS.md +++ b/evals/RESULTS.md @@ -304,3 +304,56 @@ boundary and fails if any check disappears; `explain-bug` asks a question and fails if the explanation is cut short; `vat-country` and `csv-export` fail if an existing test or an untouched module changes. +## A ninth case for long text and comments: explain-retry (crewcut 0.15.7, 2026-11-13) + +On the older cases the plain text and comment graders passed almost +everywhere, in both arms: short tasks do not invite long prose or comment +blocks. This case does. One file, a fetch with retries and no comment; the +ticket: "A new hire starts on Monday. Explain in your answer how +fetchWithRetry in src/retry.js works, step by step, and add the comments +the code needs." Fourteen graders: the code is untouched, the answer covers +six behaviours (missing two fails), no banned character in the answer or +the file, no block comment and no two comment lines in a row, every comment +gives a reason. + +| Runs passed, with / without the plugin | Fable | Opus | Sonnet | Haiku | +|---|--:|--:|--:|--:| +| Score | 1.00 / 0.79 | 0.94 / 0.70 | 0.88 / 0.73 | 0.88 / 0.70 | +| full-explanation | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 | +| plain-text-answer | 3 / 0 | 3 / 0 | 1 / 0 | 3 / 0 | +| comments-one-line | 3 / 0 | 3 / 0 | 3 / 0 | 2 / 0 | +| comments-say-why | 3 / 2 | 1 / 0 | 1 / 0 | 0 / 0 | + +Three runs per arm, 4.46 USD with the judge. The explanation is complete in +every run of both arms: the plugin's shorter answer loses nothing here. + +**Without the plugin, no model writes a plain answer and none keeps its +comments to one line**: 0 run out of 12 on each. An explanation in steps +brings long dashes and arrows, and "add the comments" brings blocks. + +**With the plugin, two weak spots.** Sonnet still puts a long dash in its +answer in 2 runs out of 3: the plain text rule holds on short answers and +gives way on a long one. And `comments-say-why` passes in 5 runs out of 12 +with the plugin. The comments were read: most give a reason ("4xx is the +caller's fault, so retrying cannot help"), and one per file names the +mechanism instead ("Exponential backoff, capped at 5s"). The grader asks +that every comment give a reason, so one such line fails the file. That is +what "why only" says; the rule is not yet followed to the letter. + +A first launch of this run found no `claude` command for any model: the +Claude Code binary was being rewritten on disk at that minute, at the same +version, as during the benchmark earlier in the day. No session had +started; the run was launched again. + +## Nine cases (crewcut 0.15.7, 2026-11-13) + +| Model | Score with | without | Turns with | without | Session cost with | without | +|---|--:|--:|--:|--:|--:|--:| +| Fable 5.2 | 0.98 | 0.88 | 4.3 | 5.7 | 0.203 USD | 0.192 USD | +| Opus 5.5 | 0.99 | 0.85 | 4.7 | 5.4 | 0.090 USD | 0.087 USD | +| Sonnet 5.5 | 0.98 | 0.86 | 4.4 | 5.3 | 0.044 USD | 0.041 USD | +| Haiku 5.5 | 0.97 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD | + +As for the eight-case table, this joins measures of the same day on the +same plugin code: the eight cases and `explain-retry`. 54 runs per model, +none in error. diff --git a/evals/explain-retry/case.yaml b/evals/explain-retry/case.yaml new file mode 100644 index 0000000..90e139e --- /dev/null +++ b/evals/explain-retry/case.yaml @@ -0,0 +1,73 @@ +schema_version: "1.0" +name: explain-retry +description: "A request for a long explanation and for comments, where a model reaches for arrows, long dashes, emoji and comment blocks. Expected: a full explanation in plain text, few one-line comments that give a reason, the code untouched." +context: + scaffold_script: scaffold.sh +execution: + prompt: A new hire starts on Monday. Explain in your answer how fetchWithRetry in src/retry.js works, step by step, and add the comments the code needs. + max_turns: 10 + timeout_seconds: 300 + allowed_tools: [Read, Edit, Grep, Glob, Skill] +runs: 3 +graders: + - type: file_exists + name: no-new-file + path: "**/*" + exists: false + - type: regex + name: keeps-attempts + target: { source: file, path: src/retry.js } + pattern: 'attempts = 4' + match: contains + - type: regex + name: keeps-status-check + target: { source: file, path: src/retry.js } + pattern: 'if \(response\.status < 500\) return response;' + match: contains + - type: regex + name: keeps-abort + target: { source: file, path: src/retry.js } + pattern: 'if \(error\.name === ''AbortError''\) throw error;' + match: contains + - type: regex + name: keeps-backoff + target: { source: file, path: src/retry.js } + pattern: 'delay = Math\.min\(delay \* 2, 5000\);' + match: contains + - type: regex + name: plain-text-answer + pattern: '[\u2013\u2014\u2018\u2019\u201C\u201D\u00A0\u202F\u2190-\u21FF\u2600-\u27BF\uD83C-\uD83E]' + match: not_contains + - type: regex + name: plain-text-in-code + target: { source: file, path: src/retry.js } + pattern: '[\u2013\u2014\u2018\u2019\u201C\u201D\u00A0\u202F\u2190-\u21FF\u2600-\u27BF\uD83C-\uD83E]' + match: not_contains + - type: regex + name: no-crewcut-in-code + target: { source: file, path: src/retry.js } + pattern: '[Cc]rewcut' + match: not_contains + - type: regex + name: comments-one-line + target: { source: file, path: src/retry.js } + pattern: '/\*|//[^\n]*\n\s*//' + match: not_contains + - type: llm + name: comments-say-why + focus: { source: file, path: src/retry.js } + criteria: > + Every comment in the file gives a reason or a constraint the code + cannot show. No comment restates what the next line does, and none + names a tool or an assistant. A file with no comment fails: the + request asked for the comments the code needs. + - type: llm + name: full-explanation + criteria: > + The final message explains all of these: it tries up to four times by + default; a response with a status below 500, 4xx included, is returned + at once and not retried; a 5xx response or a network error is retried; + an aborted request is rethrown at once; the wait starts at 200 ms, + doubles each time, is capped at 5 seconds and gets a small random + extra; after the last attempt the last error is thrown. Missing two or + more of these fails. diff --git a/evals/explain-retry/scaffold.sh b/evals/explain-retry/scaffold.sh new file mode 100755 index 0000000..0eebb74 --- /dev/null +++ b/evals/explain-retry/scaffold.sh @@ -0,0 +1,33 @@ +#!/usr/bin/env bash +set -eu +cat > package.json <<'EOF' +{ + "name": "demo-http-client", + "version": "1.0.0", + "private": true, + "type": "module", + "engines": { "node": ">=22" } +} +EOF +mkdir -p src +cat > src/retry.js <<'EOF' +export async function fetchWithRetry(url, options = {}, attempts = 4) { + let delay = 200; + let lastError; + for (let attempt = 1; attempt <= attempts; attempt++) { + try { + const response = await fetch(url, options); + if (response.status < 500) return response; + lastError = new Error(`HTTP ${response.status}`); + } catch (error) { + if (error.name === 'AbortError') throw error; + lastError = error; + } + if (attempt < attempts) { + await new Promise((resolve) => setTimeout(resolve, delay + Math.random() * 100)); + delay = Math.min(delay * 2, 5000); + } + } + throw lastError; +} +EOF diff --git a/skills/crewcut-gain/SKILL.md b/skills/crewcut-gain/SKILL.md index 2749ee4..71c9a43 100644 --- a/skills/crewcut-gain/SKILL.md +++ b/skills/crewcut-gain/SKILL.md @@ -21,13 +21,13 @@ crewcut gain measured on public repos and eval cases, not on t Next.js boilerplate, Opus 5.5 18 % 45 % 40 % 33 % 21/21 controls on Haiku, LOC: caveman 104 % and 85 %, yagni prompt 112 % and 71 % - eval cases (eight, three runs each, same model with and without) + eval cases (nine, three runs each, same model with and without) score with / without turns with / without session cost with / without - Fable 5.2 0.98 / 0.89 4.3 / 6.0 -28 % 0.197 / 0.189 USD +4 % - Opus 5.5 0.99 / 0.86 4.8 / 5.7 -16 % 0.089 / 0.086 USD +3 % - Sonnet 5.5 0.99 / 0.87 4.4 / 5.6 -21 % 0.043 / 0.040 USD +8 % - Haiku 5.5 0.98 / 0.86 5.3 / 6.4 -17 % 0.003 / 0.003 USD 0 % + Fable 5.2 0.98 / 0.88 4.3 / 5.7 -25 % 0.203 / 0.192 USD +6 % + Opus 5.5 0.99 / 0.85 4.7 / 5.4 -13 % 0.090 / 0.087 USD +3 % + Sonnet 5.5 0.98 / 0.86 4.4 / 5.3 -17 % 0.044 / 0.041 USD +7 % + Haiku 5.5 0.97 / 0.84 5.2 / 6.0 -13 % 0.003 / 0.003 USD 0 % This repo has no baseline: the version you did not build was never written, so no per-repo saving can be honestly computed.