diff --git a/CHANGELOG.md b/CHANGELOG.md index acd7155..4d69a5c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,14 @@ follows Keep a Changelog and the project follows Semantic Versioning. ## [Unreleased] +### Changed + +- Eval tables on at least six runs per case instead of three, since three + did not separate 0.97 from 1.00: Fable 5.2 1.00 against 0.87 without the + plugin, Opus 5.5 0.99 against 0.85, Sonnet 5.5 0.98 against 0.87, Haiku + 5.5 0.99 against 0.84. The notebook lists every grader that still fails + with the plugin. + ## [0.17.0] - 2026-11-13 ### Changed diff --git a/README.fr.md b/README.fr.md index d5e19b2..e4bcc67 100644 --- a/README.fr.md +++ b/README.fr.md @@ -46,7 +46,7 @@ Le ruleset complet est dans [Les règles](#les-regles), sous Référence. ## Mesuré -

Suite d'évals, neuf cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 1,00 avec crewcut, 0,88 pour le même modèle sans plugin. Opus 5.5 : 1,00 avec, 0,84 sans. Sonnet 5.5 : 0,97 avec, 0,86 sans. Haiku 5.5 : 0,97 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro.

+

Suite d'évals, neuf cas, au moins six runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 1,00 avec crewcut, 0,87 pour le même modèle sans plugin. Opus 5.5 : 0,99 avec, 0,85 sans. Sonnet 5.5 : 0,98 avec, 0,87 sans. Haiku 5.5 : 0,99 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro.

Le benchmark de ponytail 5, crewcut 0.14.0 face à sans plugin, Opus 5.5, 39 tâches, 5 runs. Lignes de code moins 58 pour cent, intervalle à 95 pour cent de moins 68 à moins 47, mieux sur 32 tâches et pire sur 1. Information de code moins 52 pour cent. Tokens de sortie moins 56 pour cent. Coût moins 34 pour cent, intervalle de moins 44 à moins 24. Temps moins 53 pour cent. Tours moins 19 pour cent. Vérifications cachées : 85 sur 90 réussies, 86 sans le plugin. Six tâches de sécurité : 30 sur 30 dans les deux bras. Tâche de confidentialité : 0 sur 5, puis 8 sur 10 après la règle qui nomme ce qu'un changement peut exposer. Ponytail 5 a publié pour lui-même : code moins 53 pour cent, tokens de sortie moins 45, coût moins 26, temps moins 41. Deux runs à un mois d'écart, les intervalles se recouvrent.

@@ -66,15 +66,15 @@ par tâche, limites et reproduction :
Les chiffres en texte -Suite d'évals, neuf cas, trois runs chacun, Opus 5.5 comme juge. Le coût est +Suite d'évals, neuf cas, au moins six runs chacun, Opus 5.5 comme juge. Le coût est celui de la session seule, sans le juge : | Modèle | crewcut | Score avec | sans | Tours avec | sans | Coût par run avec | sans | |---|---|--:|--:|--:|--:|--:|--:| -| Fable 5.2 | 0.17.0 | 1,00 | 0,88 | 4,6 | 5,7 | 0,203 USD | 0,192 USD | -| Opus 5.5 | 0.17.0 | 1,00 | 0,84 | 4,8 | 5,4 | 0,092 USD | 0,087 USD | -| Sonnet 5.5 | 0.17.0 | 0,97 | 0,86 | 4,3 | 5,3 | 0,043 USD | 0,041 USD | -| Haiku 5.5 | 0.17.0 | 0,97 | 0,84 | 5,2 | 6,0 | 0,003 USD | 0,003 USD | +| Fable 5.2 | 0.17.0 | 1,00 | 0,87 | 4,7 | 5,5 | 0,203 USD | 0,189 USD | +| Opus 5.5 | 0.17.0 | 0,99 | 0,85 | 5,0 | 5,4 | 0,093 USD | 0,088 USD | +| Sonnet 5.5 | 0.17.0 | 0,98 | 0,87 | 4,4 | 5,3 | 0,044 USD | 0,041 USD | +| Haiku 5.5 | 0.17.0 | 0,99 | 0,84 | 5,2 | 6,0 | 0,003 USD | 0,003 USD | Le benchmark de ponytail 5, Opus 5.5, 39 tâches, 5 runs, crewcut 0.14.0 face à sans plugin : diff --git a/README.md b/README.md index cbf1b4e..147360c 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ The full ruleset is in [The rules](#the-rules), under Reference. ## Measured -

Eval suite, nine cases, three runs each. Shorter, and more often right. Fable 5.2: 1.00 with crewcut, 0.88 for the same model with no plugin. Opus 5.5: 1.00 with, 0.84 without. Sonnet 5.5: 0.97 with, 0.86 without. Haiku 5.5: 0.97 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero.

+

Eval suite, nine cases, at least six runs each. Shorter, and more often right. Fable 5.2: 1.00 with crewcut, 0.87 for the same model with no plugin. Opus 5.5: 0.99 with, 0.85 without. Sonnet 5.5: 0.98 with, 0.87 without. Haiku 5.5: 0.99 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero.

Ponytail 5's benchmark, crewcut 0.14.0 against no plugin, Opus 5.5, 39 tasks, 5 runs. Lines of code minus 58 percent, 95 percent interval minus 68 to minus 47, better on 32 tasks and worse on 1. Code information minus 52 percent. Output tokens minus 56 percent. Cost minus 34 percent, interval minus 44 to minus 24. Time minus 53 percent. Turns minus 19 percent. Hidden checks: 85 of 90 passed, 86 without the plugin. Six security tasks: 30 of 30 in both arms. Privacy task: 0 of 5, then 8 of 10 after the rule that names what a change could expose. Ponytail 5 published for itself: code minus 53 percent, output tokens minus 45, cost minus 26, time minus 41. Two runs a month apart, intervals overlap.

@@ -66,15 +66,15 @@ limits and how to reproduce:
The figures as text -Eval suite, nine cases, three runs each, Opus 5.5 as judge. The cost is the +Eval suite, nine cases, at least six runs each, Opus 5.5 as judge. The cost is the session alone, without the judge: | Model | crewcut | Score with | without | Turns with | without | Cost per run with | without | |---|---|--:|--:|--:|--:|--:|--:| -| Fable 5.2 | 0.17.0 | 1.00 | 0.88 | 4.6 | 5.7 | 0.203 USD | 0.192 USD | -| Opus 5.5 | 0.17.0 | 1.00 | 0.84 | 4.8 | 5.4 | 0.092 USD | 0.087 USD | -| Sonnet 5.5 | 0.17.0 | 0.97 | 0.86 | 4.3 | 5.3 | 0.043 USD | 0.041 USD | -| Haiku 5.5 | 0.17.0 | 0.97 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD | +| Fable 5.2 | 0.17.0 | 1.00 | 0.87 | 4.7 | 5.5 | 0.203 USD | 0.189 USD | +| Opus 5.5 | 0.17.0 | 0.99 | 0.85 | 5.0 | 5.4 | 0.093 USD | 0.088 USD | +| Sonnet 5.5 | 0.17.0 | 0.98 | 0.87 | 4.4 | 5.3 | 0.044 USD | 0.041 USD | +| Haiku 5.5 | 0.17.0 | 0.99 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD | Ponytail 5's benchmark, Opus 5.5, 39 tasks, 5 runs, crewcut 0.14.0 against no plugin: diff --git a/assets/readme/evals-fr.png b/assets/readme/evals-fr.png index 1adc37c..2cbfaae 100644 Binary files a/assets/readme/evals-fr.png and b/assets/readme/evals-fr.png differ diff --git a/assets/readme/evals.png b/assets/readme/evals.png index 21b7eb0..8cd1ef0 100644 Binary files a/assets/readme/evals.png and b/assets/readme/evals.png differ diff --git a/assets/readme/src/evals-fr.html b/assets/readme/src/evals-fr.html index c78fb68..064f047 100644 --- a/assets/readme/src/evals-fr.html +++ b/assets/readme/src/evals-fr.html @@ -9,12 +9,12 @@ .legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
-
Suite d'évals, neuf cas, trois runs chacun
+
Suite d'évals, neuf cas, au moins six runs chacun

Plus court, et plus souvent juste.

-
Fable 5.2
1,00
0,88
-
Opus 5.5
1,00
0,84
-
Sonnet 5.5
0,97
0,86
-
Haiku 5.5
0,97
0,84
+
Fable 5.2
1,00
0,87
+
Opus 5.5
0,99
0,85
+
Sonnet 5.5
0,98
0,87
+
Haiku 5.5
0,99
0,84
avec crewcutmême modèle, sans plugin
Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille :
une réponse plus courte mais fausse vaut zéro.
diff --git a/assets/readme/src/evals.html b/assets/readme/src/evals.html index e7610aa..32d4e0b 100644 --- a/assets/readme/src/evals.html +++ b/assets/readme/src/evals.html @@ -9,12 +9,12 @@ .legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
-
Eval suite, nine cases, three runs each
+
Eval suite, nine cases, at least six runs each

Shorter, and more often right.

-
Fable 5.2
1.00
0.88
-
Opus 5.5
1.00
0.84
-
Sonnet 5.5
0.97
0.86
-
Haiku 5.5
0.97
0.84
+
Fable 5.2
1.00
0.87
+
Opus 5.5
0.99
0.85
+
Sonnet 5.5
0.98
0.87
+
Haiku 5.5
0.99
0.84
with crewcutsame model, no plugin
Score is the share of graders passed. Every case grades correctness as well as size:
a shorter answer that is wrong scores zero.
diff --git a/evals/RESULTS.md b/evals/RESULTS.md index ad1dd0e..d9a731d 100644 --- a/evals/RESULTS.md +++ b/evals/RESULTS.md @@ -467,3 +467,45 @@ out of 5, Haiku `csv-export` 4 out of 5, Haiku `explain-bug` 5 out of 5. About 26 USD in all. Three Sonnet runs did not start in the first suite (the same process spawn error) and their two cases were run again. + +## The table on six runs (crewcut 0.17.0, 2026-11-13) + +The previous section showed that a suite of three runs does not separate +0.97 from 1.00. A second full suite was run, both arms, four models, and +every run measured with the 0.17.0 wording and graders is pooled: six runs +per case, nine with the plugin on Sonnet and Haiku, which had been run +twice already. + +| Model | Score with | without | Runs with | without | Turns with | without | Session cost with | without | +|---|--:|--:|--:|--:|--:|--:|--:|--:| +| Fable 5.2 | 1.00 | 0.87 | 54 | 54 | 4.7 | 5.5 | 0.203 USD | 0.189 USD | +| Opus 5.5 | 0.99 | 0.85 | 54 | 54 | 5.0 | 5.4 | 0.093 USD | 0.088 USD | +| Sonnet 5.5 | 0.98 | 0.87 | 81 | 54 | 4.4 | 5.3 | 0.044 USD | 0.041 USD | +| Haiku 5.5 | 0.99 | 0.84 | 81 | 54 | 5.2 | 6.0 | 0.003 USD | 0.003 USD | + +Runs where every grader passes, with the plugin against without: Fable 53 +out of 54 against 19 out of 54, Opus 52 against 11, Sonnet 65 out of 81 +against 15 out of 54, Haiku 73 out of 81 against 8 out of 54. + +Every grader that failed with the plugin, over all those runs: + +- Fable: `csv-export` `reads-at-most-three-files` once. +- Opus: `csv-export` `reads-at-most-three-files` once, `explain-bug` + `full-explanation` once. +- Sonnet: `explain-bug` `full-explanation` 4 times out of 9, + `url-parse` `short-answer` 3, `date-picker` `only-asked-attributes` + 3 and `short-answer` 2, `explain-retry` `comments-say-why` 2 and + `plain-text-answer` 1, `keep-validation` `honest-answer` 1, + `csv-export` `reads-at-most-three-files` 1. +- Haiku: `csv-export` without the money helper 3 times out of 9, + `date-picker` `short-answer` 2, `explain-bug` `full-explanation` 2, + `explain-retry` `comments-say-why` 1. + +Sonnet is the model with the most left to gain: its two frequent misses +are an explanation that skips the caller on `explain-bug` and a fourth or +fifth sentence on `url-parse`. + +The arm without the plugin moved by one point on three models between the +two suites, which is the same spread. 27.15 USD for the second suite. Five +Haiku runs of `keep-validation` did not start (the process spawn error) +and the case was run again in both arms. diff --git a/skills/crewcut-gain/SKILL.md b/skills/crewcut-gain/SKILL.md index 8af4e99..33fd709 100644 --- a/skills/crewcut-gain/SKILL.md +++ b/skills/crewcut-gain/SKILL.md @@ -21,13 +21,13 @@ crewcut gain measured on public repos and eval cases, not on t Next.js boilerplate, Opus 5.5 18 % 45 % 40 % 33 % 21/21 controls on Haiku, LOC: caveman 104 % and 85 %, yagni prompt 112 % and 71 % - eval cases (nine, three runs each, same model with and without) + eval cases (nine, at least six runs each, same model with and without) score with / without turns with / without session cost with / without - Fable 5.2 1.00 / 0.88 4.6 / 5.7 -19 % 0.203 / 0.192 USD +6 % - Opus 5.5 1.00 / 0.84 4.8 / 5.4 -11 % 0.092 / 0.087 USD +6 % - Sonnet 5.5 0.97 / 0.86 4.3 / 5.3 -19 % 0.043 / 0.041 USD +5 % - Haiku 5.5 0.97 / 0.84 5.2 / 6.0 -13 % 0.003 / 0.003 USD 0 % + Fable 5.2 1.00 / 0.87 4.7 / 5.5 -15 % 0.203 / 0.189 USD +8 % + Opus 5.5 0.99 / 0.85 5.0 / 5.4 -7 % 0.093 / 0.088 USD +6 % + Sonnet 5.5 0.98 / 0.87 4.4 / 5.3 -17 % 0.044 / 0.041 USD +7 % + Haiku 5.5 0.99 / 0.84 5.2 / 6.0 -13 % 0.003 / 0.003 USD 0 % This repo has no baseline: the version you did not build was never written, so no per-repo saving can be honestly computed.