diff --git a/CHANGELOG.md b/CHANGELOG.md index 5dd8007..d76eaa4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,13 @@ follows Keep a Changelog and the project follows Semantic Versioning. ## [Unreleased] +### Changed + +- The eval tables and the eval card count eight cases: the seven measured + with Opus as judge and `zod-email`, same day, same plugin code. Fable 5.2 + 0.98 against 0.89 without the plugin, Opus 5.5 0.99 against 0.86, Sonnet + 5.5 0.99 against 0.87, Haiku 5.5 0.98 against 0.86. + ## [0.15.6] - 2026-11-13 ### Added diff --git a/README.fr.md b/README.fr.md index 8fb334a..559e560 100644 --- a/README.fr.md +++ b/README.fr.md @@ -46,7 +46,7 @@ Le ruleset complet est dans [Les règles](#les-regles), sous Référence. ## Mesuré -

Suite d'évals, sept cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,98 avec crewcut, 0,87 pour le même modèle sans plugin. Opus 5.5 : 1,00 avec, 0,80 sans. Sonnet 5.5 : 0,99 avec, 0,88 sans. Haiku 5.5 : 0,98 avec, 0,87 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro.

+

Suite d'évals, huit cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,98 avec crewcut, 0,89 pour le même modèle sans plugin. Opus 5.5 : 0,99 avec, 0,86 sans. Sonnet 5.5 : 0,99 avec, 0,87 sans. Haiku 5.5 : 0,98 avec, 0,86 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro.

Le benchmark de ponytail 5, crewcut 0.14.0 face à sans plugin, Opus 5.5, 39 tâches, 5 runs. Lignes de code moins 58 pour cent, intervalle à 95 pour cent de moins 68 à moins 47, mieux sur 32 tâches et pire sur 1. Information de code moins 52 pour cent. Tokens de sortie moins 56 pour cent. Coût moins 34 pour cent, intervalle de moins 44 à moins 24. Temps moins 53 pour cent. Tours moins 19 pour cent. Vérifications cachées : 85 sur 90 réussies, 86 sans le plugin. Six tâches de sécurité : 30 sur 30 dans les deux bras. Tâche de confidentialité : 0 sur 5, puis 8 sur 10 après la règle qui nomme ce qu'un changement peut exposer. Ponytail 5 a publié pour lui-même : code moins 53 pour cent, tokens de sortie moins 45, coût moins 26, temps moins 41. Deux runs à un mois d'écart, les intervalles se recouvrent.

@@ -66,15 +66,15 @@ par tâche, limites et reproduction :
Les chiffres en texte -Suite d'évals, sept cas, trois runs chacun, Opus 5.5 comme juge. Le coût est +Suite d'évals, huit cas, trois runs chacun, Opus 5.5 comme juge. Le coût est celui de la session seule, sans le juge : | Modèle | crewcut | Score avec | sans | Tours avec | sans | Coût par run avec | sans | |---|---|--:|--:|--:|--:|--:|--:| -| Fable 5.2 | 0.15.3 | 0,98 | 0,87 | 4,4 | 6,1 | 0,202 USD | 0,193 USD | -| Opus 5.5 | 0.15.3 | 1,00 | 0,80 | 5,3 | 6,0 | 0,092 USD | 0,089 USD | -| Sonnet 5.5 | 0.15.3 | 0,99 | 0,88 | 4,6 | 6,0 | 0,045 USD | 0,042 USD | -| Haiku 5.5 | 0.15.3 | 0,98 | 0,87 | 5,6 | 6,9 | 0,003 USD | 0,003 USD | +| Fable 5.2 | 0.15.6 | 0,98 | 0,89 | 4,3 | 6,0 | 0,197 USD | 0,189 USD | +| Opus 5.5 | 0.15.6 | 0,99 | 0,86 | 4,8 | 5,7 | 0,089 USD | 0,086 USD | +| Sonnet 5.5 | 0.15.6 | 0,99 | 0,87 | 4,4 | 5,6 | 0,043 USD | 0,040 USD | +| Haiku 5.5 | 0.15.6 | 0,98 | 0,86 | 5,3 | 6,4 | 0,003 USD | 0,003 USD | Le benchmark de ponytail 5, Opus 5.5, 39 tâches, 5 runs, crewcut 0.14.0 face à sans plugin : diff --git a/README.md b/README.md index 17ee2f6..b439dd8 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ The full ruleset is in [The rules](#the-rules), under Reference. ## Measured -

Eval suite, seven cases, three runs each. Shorter, and more often right. Fable 5.2: 0.98 with crewcut, 0.87 for the same model with no plugin. Opus 5.5: 1.00 with, 0.80 without. Sonnet 5.5: 0.99 with, 0.88 without. Haiku 5.5: 0.98 with, 0.87 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero.

+

Eval suite, eight cases, three runs each. Shorter, and more often right. Fable 5.2: 0.98 with crewcut, 0.89 for the same model with no plugin. Opus 5.5: 0.99 with, 0.86 without. Sonnet 5.5: 0.99 with, 0.87 without. Haiku 5.5: 0.98 with, 0.86 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero.

Ponytail 5's benchmark, crewcut 0.14.0 against no plugin, Opus 5.5, 39 tasks, 5 runs. Lines of code minus 58 percent, 95 percent interval minus 68 to minus 47, better on 32 tasks and worse on 1. Code information minus 52 percent. Output tokens minus 56 percent. Cost minus 34 percent, interval minus 44 to minus 24. Time minus 53 percent. Turns minus 19 percent. Hidden checks: 85 of 90 passed, 86 without the plugin. Six security tasks: 30 of 30 in both arms. Privacy task: 0 of 5, then 8 of 10 after the rule that names what a change could expose. Ponytail 5 published for itself: code minus 53 percent, output tokens minus 45, cost minus 26, time minus 41. Two runs a month apart, intervals overlap.

@@ -66,15 +66,15 @@ limits and how to reproduce:
The figures as text -Eval suite, seven cases, three runs each, Opus 5.5 as judge. The cost is the +Eval suite, eight cases, three runs each, Opus 5.5 as judge. The cost is the session alone, without the judge: | Model | crewcut | Score with | without | Turns with | without | Cost per run with | without | |---|---|--:|--:|--:|--:|--:|--:| -| Fable 5.2 | 0.15.3 | 0.98 | 0.87 | 4.4 | 6.1 | 0.202 USD | 0.193 USD | -| Opus 5.5 | 0.15.3 | 1.00 | 0.80 | 5.3 | 6.0 | 0.092 USD | 0.089 USD | -| Sonnet 5.5 | 0.15.3 | 0.99 | 0.88 | 4.6 | 6.0 | 0.045 USD | 0.042 USD | -| Haiku 5.5 | 0.15.3 | 0.98 | 0.87 | 5.6 | 6.9 | 0.003 USD | 0.003 USD | +| Fable 5.2 | 0.15.6 | 0.98 | 0.89 | 4.3 | 6.0 | 0.197 USD | 0.189 USD | +| Opus 5.5 | 0.15.6 | 0.99 | 0.86 | 4.8 | 5.7 | 0.089 USD | 0.086 USD | +| Sonnet 5.5 | 0.15.6 | 0.99 | 0.87 | 4.4 | 5.6 | 0.043 USD | 0.040 USD | +| Haiku 5.5 | 0.15.6 | 0.98 | 0.86 | 5.3 | 6.4 | 0.003 USD | 0.003 USD | Ponytail 5's benchmark, Opus 5.5, 39 tasks, 5 runs, crewcut 0.14.0 against no plugin: diff --git a/assets/readme/evals-fr.png b/assets/readme/evals-fr.png index 2bca8a6..b7591d4 100644 Binary files a/assets/readme/evals-fr.png and b/assets/readme/evals-fr.png differ diff --git a/assets/readme/evals.png b/assets/readme/evals.png index 24767a3..0ee9a84 100644 Binary files a/assets/readme/evals.png and b/assets/readme/evals.png differ diff --git a/assets/readme/src/evals-fr.html b/assets/readme/src/evals-fr.html index 1048b18..102822f 100644 --- a/assets/readme/src/evals-fr.html +++ b/assets/readme/src/evals-fr.html @@ -9,12 +9,12 @@ .legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
-
Suite d'évals, sept cas, trois runs chacun
+
Suite d'évals, huit cas, trois runs chacun

Plus court, et plus souvent juste.

-
Fable 5.2
0,98
0,87
-
Opus 5.5
1,00
0,80
-
Sonnet 5.5
0,99
0,88
-
Haiku 5.5
0,98
0,87
+
Fable 5.2
0,98
0,89
+
Opus 5.5
0,99
0,86
+
Sonnet 5.5
0,99
0,87
+
Haiku 5.5
0,98
0,86
avec crewcutmême modèle, sans plugin
Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille :
une réponse plus courte mais fausse vaut zéro.
diff --git a/assets/readme/src/evals.html b/assets/readme/src/evals.html index 84f6401..cbd7990 100644 --- a/assets/readme/src/evals.html +++ b/assets/readme/src/evals.html @@ -9,12 +9,12 @@ .legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
-
Eval suite, seven cases, three runs each
+
Eval suite, eight cases, three runs each

Shorter, and more often right.

-
Fable 5.2
0.98
0.87
-
Opus 5.5
1.00
0.80
-
Sonnet 5.5
0.99
0.88
-
Haiku 5.5
0.98
0.87
+
Fable 5.2
0.98
0.89
+
Opus 5.5
0.99
0.86
+
Sonnet 5.5
0.99
0.87
+
Haiku 5.5
0.98
0.86
with crewcutsame model, no plugin
Score is the share of graders passed. Every case grades correctness as well as size:
a shorter answer that is wrong scores zero.
diff --git a/evals/RESULTS.md b/evals/RESULTS.md index 9a1a7df..6a0eb12 100644 --- a/evals/RESULTS.md +++ b/evals/RESULTS.md @@ -206,6 +206,32 @@ in every trace. Check the trace before reading an ablation of the map. The table of the previous section is over the seven older cases; the next full measure counts eight. +## Eight cases (crewcut 0.15.6, 2026-11-13) + +| Model | Score with | without | Turns with | without | Session cost with | without | +|---|--:|--:|--:|--:|--:|--:| +| Fable 5.2 | 0.98 | 0.89 | 4.3 | 6.0 | 0.197 USD | 0.189 USD | +| Opus 5.5 | 0.99 | 0.86 | 4.8 | 5.7 | 0.089 USD | 0.086 USD | +| Sonnet 5.5 | 0.99 | 0.87 | 4.4 | 5.6 | 0.043 USD | 0.040 USD | +| Haiku 5.5 | 0.98 | 0.86 | 5.3 | 6.4 | 0.003 USD | 0.003 USD | + +This table is not a new run. It joins the seven cases of the section before +last and `zod-email`, measured two hours apart on the same day, with the +same judge, the same Claude Code and the same plugin code: between crewcut +0.15.3 and 0.15.6 only the version string, the examples, the README and the +eval cases changed, none of which reaches a session. One case was run +again: `explain-bug` on Opus, whose first measure had lost a run without +the plugin to a process spawn error; with three runs its score without the +plugin is higher, which is most of the move from 0.80 to 0.86 on that row. +48 runs per model, none in error, none out of usage. + +With the plugin, what still fails: on Fable, `date-picker` `short-answer` +twice and `shared-bug` `root-cause-stated` three times (the fix is in the +shared function in every run, the judge wants the cause said more plainly); +on Opus and Sonnet, `explain-bug` `full-explanation` once each; on Haiku, +`date-picker` `short-answer` three times and one banned character in an +answer. + What it says: - The savings show up with the size of the project. `csv-export` is twenty diff --git a/skills/crewcut-gain/SKILL.md b/skills/crewcut-gain/SKILL.md index 1f3daf3..2749ee4 100644 --- a/skills/crewcut-gain/SKILL.md +++ b/skills/crewcut-gain/SKILL.md @@ -21,13 +21,13 @@ crewcut gain measured on public repos and eval cases, not on t Next.js boilerplate, Opus 5.5 18 % 45 % 40 % 33 % 21/21 controls on Haiku, LOC: caveman 104 % and 85 %, yagni prompt 112 % and 71 % - eval cases (seven, three runs each, same model with and without) + eval cases (eight, three runs each, same model with and without) score with / without turns with / without session cost with / without - Fable 5.2 0.98 / 0.87 4.4 / 6.1 -28 % 0.202 / 0.193 USD +5 % - Opus 5.5 1.00 / 0.80 5.3 / 6.0 -12 % 0.092 / 0.089 USD +3 % - Sonnet 5.5 0.99 / 0.88 4.6 / 6.0 -23 % 0.045 / 0.042 USD +7 % - Haiku 5.5 0.98 / 0.87 5.6 / 6.9 -19 % 0.003 / 0.003 USD 0 % + Fable 5.2 0.98 / 0.89 4.3 / 6.0 -28 % 0.197 / 0.189 USD +4 % + Opus 5.5 0.99 / 0.86 4.8 / 5.7 -16 % 0.089 / 0.086 USD +3 % + Sonnet 5.5 0.99 / 0.87 4.4 / 5.6 -21 % 0.043 / 0.040 USD +8 % + Haiku 5.5 0.98 / 0.86 5.3 / 6.4 -17 % 0.003 / 0.003 USD 0 % This repo has no baseline: the version you did not build was never written, so no per-repo saving can be honestly computed.