Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,13 @@ follows Keep a Changelog and the project follows Semantic Versioning.

## [Unreleased]

### Changed

- The eval tables and the eval card count eight cases: the seven measured
with Opus as judge and `zod-email`, same day, same plugin code. Fable 5.2
0.98 against 0.89 without the plugin, Opus 5.5 0.99 against 0.86, Sonnet
5.5 0.99 against 0.87, Haiku 5.5 0.98 against 0.86.

## [0.15.6] - 2026-11-13

### Added
Expand Down
12 changes: 6 additions & 6 deletions README.fr.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Le ruleset complet est dans [Les règles](#les-regles), sous Référence.

## Mesuré

<p align="center"><img src="assets/readme/evals-fr.png" width="880" alt="Suite d'évals, sept cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,98 avec crewcut, 0,87 pour le même modèle sans plugin. Opus 5.5 : 1,00 avec, 0,80 sans. Sonnet 5.5 : 0,99 avec, 0,88 sans. Haiku 5.5 : 0,98 avec, 0,87 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro."></p>
<p align="center"><img src="assets/readme/evals-fr.png" width="880" alt="Suite d'évals, huit cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,98 avec crewcut, 0,89 pour le même modèle sans plugin. Opus 5.5 : 0,99 avec, 0,86 sans. Sonnet 5.5 : 0,99 avec, 0,87 sans. Haiku 5.5 : 0,98 avec, 0,86 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro."></p>

<p align="center"><img src="assets/readme/pt5-fr.png" width="880" alt="Le benchmark de ponytail 5, crewcut 0.14.0 face à sans plugin, Opus 5.5, 39 tâches, 5 runs. Lignes de code moins 58 pour cent, intervalle à 95 pour cent de moins 68 à moins 47, mieux sur 32 tâches et pire sur 1. Information de code moins 52 pour cent. Tokens de sortie moins 56 pour cent. Coût moins 34 pour cent, intervalle de moins 44 à moins 24. Temps moins 53 pour cent. Tours moins 19 pour cent. Vérifications cachées : 85 sur 90 réussies, 86 sans le plugin. Six tâches de sécurité : 30 sur 30 dans les deux bras. Tâche de confidentialité : 0 sur 5, puis 8 sur 10 après la règle qui nomme ce qu'un changement peut exposer. Ponytail 5 a publié pour lui-même : code moins 53 pour cent, tokens de sortie moins 45, coût moins 26, temps moins 41. Deux runs à un mois d'écart, les intervalles se recouvrent."></p>

Expand All @@ -66,15 +66,15 @@ par tâche, limites et reproduction :
<details>
<summary>Les chiffres en texte</summary>

Suite d'évals, sept cas, trois runs chacun, Opus 5.5 comme juge. Le coût est
Suite d'évals, huit cas, trois runs chacun, Opus 5.5 comme juge. Le coût est
celui de la session seule, sans le juge :

| Modèle | crewcut | Score avec | sans | Tours avec | sans | Coût par run avec | sans |
|---|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.15.3 | 0,98 | 0,87 | 4,4 | 6,1 | 0,202 USD | 0,193 USD |
| Opus 5.5 | 0.15.3 | 1,00 | 0,80 | 5,3 | 6,0 | 0,092 USD | 0,089 USD |
| Sonnet 5.5 | 0.15.3 | 0,99 | 0,88 | 4,6 | 6,0 | 0,045 USD | 0,042 USD |
| Haiku 5.5 | 0.15.3 | 0,98 | 0,87 | 5,6 | 6,9 | 0,003 USD | 0,003 USD |
| Fable 5.2 | 0.15.6 | 0,98 | 0,89 | 4,3 | 6,0 | 0,197 USD | 0,189 USD |
| Opus 5.5 | 0.15.6 | 0,99 | 0,86 | 4,8 | 5,7 | 0,089 USD | 0,086 USD |
| Sonnet 5.5 | 0.15.6 | 0,99 | 0,87 | 4,4 | 5,6 | 0,043 USD | 0,040 USD |
| Haiku 5.5 | 0.15.6 | 0,98 | 0,86 | 5,3 | 6,4 | 0,003 USD | 0,003 USD |

Le benchmark de ponytail 5, Opus 5.5, 39 tâches, 5 runs, crewcut 0.14.0
face à sans plugin :
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ The full ruleset is in [The rules](#the-rules), under Reference.

## Measured

<p align="center"><img src="assets/readme/evals.png" width="880" alt="Eval suite, seven cases, three runs each. Shorter, and more often right. Fable 5.2: 0.98 with crewcut, 0.87 for the same model with no plugin. Opus 5.5: 1.00 with, 0.80 without. Sonnet 5.5: 0.99 with, 0.88 without. Haiku 5.5: 0.98 with, 0.87 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero."></p>
<p align="center"><img src="assets/readme/evals.png" width="880" alt="Eval suite, eight cases, three runs each. Shorter, and more often right. Fable 5.2: 0.98 with crewcut, 0.89 for the same model with no plugin. Opus 5.5: 0.99 with, 0.86 without. Sonnet 5.5: 0.99 with, 0.87 without. Haiku 5.5: 0.98 with, 0.86 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero."></p>

<p align="center"><img src="assets/readme/pt5.png" width="880" alt="Ponytail 5's benchmark, crewcut 0.14.0 against no plugin, Opus 5.5, 39 tasks, 5 runs. Lines of code minus 58 percent, 95 percent interval minus 68 to minus 47, better on 32 tasks and worse on 1. Code information minus 52 percent. Output tokens minus 56 percent. Cost minus 34 percent, interval minus 44 to minus 24. Time minus 53 percent. Turns minus 19 percent. Hidden checks: 85 of 90 passed, 86 without the plugin. Six security tasks: 30 of 30 in both arms. Privacy task: 0 of 5, then 8 of 10 after the rule that names what a change could expose. Ponytail 5 published for itself: code minus 53 percent, output tokens minus 45, cost minus 26, time minus 41. Two runs a month apart, intervals overlap."></p>

Expand All @@ -66,15 +66,15 @@ limits and how to reproduce:
<details>
<summary>The figures as text</summary>

Eval suite, seven cases, three runs each, Opus 5.5 as judge. The cost is the
Eval suite, eight cases, three runs each, Opus 5.5 as judge. The cost is the
session alone, without the judge:

| Model | crewcut | Score with | without | Turns with | without | Cost per run with | without |
|---|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.15.3 | 0.98 | 0.87 | 4.4 | 6.1 | 0.202 USD | 0.193 USD |
| Opus 5.5 | 0.15.3 | 1.00 | 0.80 | 5.3 | 6.0 | 0.092 USD | 0.089 USD |
| Sonnet 5.5 | 0.15.3 | 0.99 | 0.88 | 4.6 | 6.0 | 0.045 USD | 0.042 USD |
| Haiku 5.5 | 0.15.3 | 0.98 | 0.87 | 5.6 | 6.9 | 0.003 USD | 0.003 USD |
| Fable 5.2 | 0.15.6 | 0.98 | 0.89 | 4.3 | 6.0 | 0.197 USD | 0.189 USD |
| Opus 5.5 | 0.15.6 | 0.99 | 0.86 | 4.8 | 5.7 | 0.089 USD | 0.086 USD |
| Sonnet 5.5 | 0.15.6 | 0.99 | 0.87 | 4.4 | 5.6 | 0.043 USD | 0.040 USD |
| Haiku 5.5 | 0.15.6 | 0.98 | 0.86 | 5.3 | 6.4 | 0.003 USD | 0.003 USD |

Ponytail 5's benchmark, Opus 5.5, 39 tasks, 5 runs, crewcut 0.14.0 against
no plugin:
Expand Down
Binary file modified assets/readme/evals-fr.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/readme/evals.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions assets/readme/src/evals-fr.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@
.legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
</style>
<div class="card"><div class="pole"></div>
<div class="eyebrow">Suite d'évals, sept cas, trois runs chacun</div>
<div class="eyebrow">Suite d'évals, huit cas, trois runs chacun</div>
<h1>Plus court, et <em>plus souvent juste.</em></h1>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:87%">0,87</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:100%">1,00</div><div class="bar o" style="width:80%">0,80</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:99%">0,99</div><div class="bar o" style="width:88%">0,88</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:87%">0,87</div></div></div>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:89%">0,89</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:99%">0,99</div><div class="bar o" style="width:86%">0,86</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:99%">0,99</div><div class="bar o" style="width:87%">0,87</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:86%">0,86</div></div></div>
<div class="legend"><span><i style="background:#E34948"></i>avec crewcut</span><span><i style="background:#2A3945"></i>même modèle, sans plugin</span></div>
<div class="foot">Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille :<br>une réponse plus courte mais fausse vaut zéro.</div>
</div>
Expand Down
10 changes: 5 additions & 5 deletions assets/readme/src/evals.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@
.legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
</style>
<div class="card"><div class="pole"></div>
<div class="eyebrow">Eval suite, seven cases, three runs each</div>
<div class="eyebrow">Eval suite, eight cases, three runs each</div>
<h1>Shorter, and <em>more often right.</em></h1>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:87%">0.87</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:100%">1.00</div><div class="bar o" style="width:80%">0.80</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:99%">0.99</div><div class="bar o" style="width:88%">0.88</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:87%">0.87</div></div></div>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:89%">0.89</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:99%">0.99</div><div class="bar o" style="width:86%">0.86</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:99%">0.99</div><div class="bar o" style="width:87%">0.87</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:86%">0.86</div></div></div>
<div class="legend"><span><i style="background:#E34948"></i>with crewcut</span><span><i style="background:#2A3945"></i>same model, no plugin</span></div>
<div class="foot">Score is the share of graders passed. Every case grades correctness as well as size:<br>a shorter answer that is wrong scores zero.</div>
</div>
Expand Down
26 changes: 26 additions & 0 deletions evals/RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,32 @@ in every trace. Check the trace before reading an ablation of the map.
The table of the previous section is over the seven older cases; the next
full measure counts eight.

## Eight cases (crewcut 0.15.6, 2026-11-13)

| Model | Score with | without | Turns with | without | Session cost with | without |
|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.98 | 0.89 | 4.3 | 6.0 | 0.197 USD | 0.189 USD |
| Opus 5.5 | 0.99 | 0.86 | 4.8 | 5.7 | 0.089 USD | 0.086 USD |
| Sonnet 5.5 | 0.99 | 0.87 | 4.4 | 5.6 | 0.043 USD | 0.040 USD |
| Haiku 5.5 | 0.98 | 0.86 | 5.3 | 6.4 | 0.003 USD | 0.003 USD |

This table is not a new run. It joins the seven cases of the section before
last and `zod-email`, measured two hours apart on the same day, with the
same judge, the same Claude Code and the same plugin code: between crewcut
0.15.3 and 0.15.6 only the version string, the examples, the README and the
eval cases changed, none of which reaches a session. One case was run
again: `explain-bug` on Opus, whose first measure had lost a run without
the plugin to a process spawn error; with three runs its score without the
plugin is higher, which is most of the move from 0.80 to 0.86 on that row.
48 runs per model, none in error, none out of usage.

With the plugin, what still fails: on Fable, `date-picker` `short-answer`
twice and `shared-bug` `root-cause-stated` three times (the fix is in the
shared function in every run, the judge wants the cause said more plainly);
on Opus and Sonnet, `explain-bug` `full-explanation` once each; on Haiku,
`date-picker` `short-answer` three times and one banned character in an
answer.

What it says:

- The savings show up with the size of the project. `csv-export` is twenty
Expand Down
10 changes: 5 additions & 5 deletions skills/crewcut-gain/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ crewcut gain measured on public repos and eval cases, not on t
Next.js boilerplate, Opus 5.5 18 % 45 % 40 % 33 % 21/21
controls on Haiku, LOC: caveman 104 % and 85 %, yagni prompt 112 % and 71 %

eval cases (seven, three runs each, same model with and without)
eval cases (eight, three runs each, same model with and without)

score with / without turns with / without session cost with / without
Fable 5.2 0.98 / 0.87 4.4 / 6.1 -28 % 0.202 / 0.193 USD +5 %
Opus 5.5 1.00 / 0.80 5.3 / 6.0 -12 % 0.092 / 0.089 USD +3 %
Sonnet 5.5 0.99 / 0.88 4.6 / 6.0 -23 % 0.045 / 0.042 USD +7 %
Haiku 5.5 0.98 / 0.87 5.6 / 6.9 -19 % 0.003 / 0.003 USD 0 %
Fable 5.2 0.98 / 0.89 4.3 / 6.0 -28 % 0.197 / 0.189 USD +4 %
Opus 5.5 0.99 / 0.86 4.8 / 5.7 -16 % 0.089 / 0.086 USD +3 %
Sonnet 5.5 0.99 / 0.87 4.4 / 5.6 -21 % 0.043 / 0.040 USD +8 %
Haiku 5.5 0.98 / 0.86 5.3 / 6.4 -17 % 0.003 / 0.003 USD 0 %

This repo has no baseline: the version you did not build was never written,
so no per-repo saving can be honestly computed.
Expand Down
Loading