Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ follows Keep a Changelog and the project follows Semantic Versioning.

## [Unreleased]

### Changed

- Eval tables on at least six runs per case instead of three, since three
did not separate 0.97 from 1.00: Fable 5.2 1.00 against 0.87 without the
plugin, Opus 5.5 0.99 against 0.85, Sonnet 5.5 0.98 against 0.87, Haiku
5.5 0.99 against 0.84. The notebook lists every grader that still fails
with the plugin.

## [0.17.0] - 2026-11-13

### Changed
Expand Down
12 changes: 6 additions & 6 deletions README.fr.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Le ruleset complet est dans [Les règles](#les-regles), sous Référence.

## Mesuré

<p align="center"><img src="assets/readme/evals-fr.png" width="880" alt="Suite d'évals, neuf cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 1,00 avec crewcut, 0,88 pour le même modèle sans plugin. Opus 5.5 : 1,00 avec, 0,84 sans. Sonnet 5.5 : 0,97 avec, 0,86 sans. Haiku 5.5 : 0,97 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro."></p>
<p align="center"><img src="assets/readme/evals-fr.png" width="880" alt="Suite d'évals, neuf cas, au moins six runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 1,00 avec crewcut, 0,87 pour le même modèle sans plugin. Opus 5.5 : 0,99 avec, 0,85 sans. Sonnet 5.5 : 0,98 avec, 0,87 sans. Haiku 5.5 : 0,99 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro."></p>

<p align="center"><img src="assets/readme/pt5-fr.png" width="880" alt="Le benchmark de ponytail 5, crewcut 0.14.0 face à sans plugin, Opus 5.5, 39 tâches, 5 runs. Lignes de code moins 58 pour cent, intervalle à 95 pour cent de moins 68 à moins 47, mieux sur 32 tâches et pire sur 1. Information de code moins 52 pour cent. Tokens de sortie moins 56 pour cent. Coût moins 34 pour cent, intervalle de moins 44 à moins 24. Temps moins 53 pour cent. Tours moins 19 pour cent. Vérifications cachées : 85 sur 90 réussies, 86 sans le plugin. Six tâches de sécurité : 30 sur 30 dans les deux bras. Tâche de confidentialité : 0 sur 5, puis 8 sur 10 après la règle qui nomme ce qu'un changement peut exposer. Ponytail 5 a publié pour lui-même : code moins 53 pour cent, tokens de sortie moins 45, coût moins 26, temps moins 41. Deux runs à un mois d'écart, les intervalles se recouvrent."></p>

Expand All @@ -66,15 +66,15 @@ par tâche, limites et reproduction :
<details>
<summary>Les chiffres en texte</summary>

Suite d'évals, neuf cas, trois runs chacun, Opus 5.5 comme juge. Le coût est
Suite d'évals, neuf cas, au moins six runs chacun, Opus 5.5 comme juge. Le coût est
celui de la session seule, sans le juge :

| Modèle | crewcut | Score avec | sans | Tours avec | sans | Coût par run avec | sans |
|---|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.17.0 | 1,00 | 0,88 | 4,6 | 5,7 | 0,203 USD | 0,192 USD |
| Opus 5.5 | 0.17.0 | 1,00 | 0,84 | 4,8 | 5,4 | 0,092 USD | 0,087 USD |
| Sonnet 5.5 | 0.17.0 | 0,97 | 0,86 | 4,3 | 5,3 | 0,043 USD | 0,041 USD |
| Haiku 5.5 | 0.17.0 | 0,97 | 0,84 | 5,2 | 6,0 | 0,003 USD | 0,003 USD |
| Fable 5.2 | 0.17.0 | 1,00 | 0,87 | 4,7 | 5,5 | 0,203 USD | 0,189 USD |
| Opus 5.5 | 0.17.0 | 0,99 | 0,85 | 5,0 | 5,4 | 0,093 USD | 0,088 USD |
| Sonnet 5.5 | 0.17.0 | 0,98 | 0,87 | 4,4 | 5,3 | 0,044 USD | 0,041 USD |
| Haiku 5.5 | 0.17.0 | 0,99 | 0,84 | 5,2 | 6,0 | 0,003 USD | 0,003 USD |

Le benchmark de ponytail 5, Opus 5.5, 39 tâches, 5 runs, crewcut 0.14.0
face à sans plugin :
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ The full ruleset is in [The rules](#the-rules), under Reference.

## Measured

<p align="center"><img src="assets/readme/evals.png" width="880" alt="Eval suite, nine cases, three runs each. Shorter, and more often right. Fable 5.2: 1.00 with crewcut, 0.88 for the same model with no plugin. Opus 5.5: 1.00 with, 0.84 without. Sonnet 5.5: 0.97 with, 0.86 without. Haiku 5.5: 0.97 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero."></p>
<p align="center"><img src="assets/readme/evals.png" width="880" alt="Eval suite, nine cases, at least six runs each. Shorter, and more often right. Fable 5.2: 1.00 with crewcut, 0.87 for the same model with no plugin. Opus 5.5: 0.99 with, 0.85 without. Sonnet 5.5: 0.98 with, 0.87 without. Haiku 5.5: 0.99 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero."></p>

<p align="center"><img src="assets/readme/pt5.png" width="880" alt="Ponytail 5's benchmark, crewcut 0.14.0 against no plugin, Opus 5.5, 39 tasks, 5 runs. Lines of code minus 58 percent, 95 percent interval minus 68 to minus 47, better on 32 tasks and worse on 1. Code information minus 52 percent. Output tokens minus 56 percent. Cost minus 34 percent, interval minus 44 to minus 24. Time minus 53 percent. Turns minus 19 percent. Hidden checks: 85 of 90 passed, 86 without the plugin. Six security tasks: 30 of 30 in both arms. Privacy task: 0 of 5, then 8 of 10 after the rule that names what a change could expose. Ponytail 5 published for itself: code minus 53 percent, output tokens minus 45, cost minus 26, time minus 41. Two runs a month apart, intervals overlap."></p>

Expand All @@ -66,15 +66,15 @@ limits and how to reproduce:
<details>
<summary>The figures as text</summary>

Eval suite, nine cases, three runs each, Opus 5.5 as judge. The cost is the
Eval suite, nine cases, at least six runs each, Opus 5.5 as judge. The cost is the
session alone, without the judge:

| Model | crewcut | Score with | without | Turns with | without | Cost per run with | without |
|---|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.17.0 | 1.00 | 0.88 | 4.6 | 5.7 | 0.203 USD | 0.192 USD |
| Opus 5.5 | 0.17.0 | 1.00 | 0.84 | 4.8 | 5.4 | 0.092 USD | 0.087 USD |
| Sonnet 5.5 | 0.17.0 | 0.97 | 0.86 | 4.3 | 5.3 | 0.043 USD | 0.041 USD |
| Haiku 5.5 | 0.17.0 | 0.97 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD |
| Fable 5.2 | 0.17.0 | 1.00 | 0.87 | 4.7 | 5.5 | 0.203 USD | 0.189 USD |
| Opus 5.5 | 0.17.0 | 0.99 | 0.85 | 5.0 | 5.4 | 0.093 USD | 0.088 USD |
| Sonnet 5.5 | 0.17.0 | 0.98 | 0.87 | 4.4 | 5.3 | 0.044 USD | 0.041 USD |
| Haiku 5.5 | 0.17.0 | 0.99 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD |

Ponytail 5's benchmark, Opus 5.5, 39 tasks, 5 runs, crewcut 0.14.0 against
no plugin:
Expand Down
Binary file modified assets/readme/evals-fr.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/readme/evals.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions assets/readme/src/evals-fr.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@
.legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
</style>
<div class="card"><div class="pole"></div>
<div class="eyebrow">Suite d'évals, neuf cas, trois runs chacun</div>
<div class="eyebrow">Suite d'évals, neuf cas, au moins six runs chacun</div>
<h1>Plus court, et <em>plus souvent juste.</em></h1>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:100%">1,00</div><div class="bar o" style="width:88%">0,88</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:100%">1,00</div><div class="bar o" style="width:84%">0,84</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:97%">0,97</div><div class="bar o" style="width:86%">0,86</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:97%">0,97</div><div class="bar o" style="width:84%">0,84</div></div></div>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:100%">1,00</div><div class="bar o" style="width:87%">0,87</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:99%">0,99</div><div class="bar o" style="width:85%">0,85</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:87%">0,87</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:99%">0,99</div><div class="bar o" style="width:84%">0,84</div></div></div>
<div class="legend"><span><i style="background:#E34948"></i>avec crewcut</span><span><i style="background:#2A3945"></i>même modèle, sans plugin</span></div>
<div class="foot">Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille :<br>une réponse plus courte mais fausse vaut zéro.</div>
</div>
Expand Down
10 changes: 5 additions & 5 deletions assets/readme/src/evals.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@
.legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
</style>
<div class="card"><div class="pole"></div>
<div class="eyebrow">Eval suite, nine cases, three runs each</div>
<div class="eyebrow">Eval suite, nine cases, at least six runs each</div>
<h1>Shorter, and <em>more often right.</em></h1>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:100%">1.00</div><div class="bar o" style="width:88%">0.88</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:100%">1.00</div><div class="bar o" style="width:84%">0.84</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:97%">0.97</div><div class="bar o" style="width:86%">0.86</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:97%">0.97</div><div class="bar o" style="width:84%">0.84</div></div></div>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:100%">1.00</div><div class="bar o" style="width:87%">0.87</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:99%">0.99</div><div class="bar o" style="width:85%">0.85</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:87%">0.87</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:99%">0.99</div><div class="bar o" style="width:84%">0.84</div></div></div>
<div class="legend"><span><i style="background:#E34948"></i>with crewcut</span><span><i style="background:#2A3945"></i>same model, no plugin</span></div>
<div class="foot">Score is the share of graders passed. Every case grades correctness as well as size:<br>a shorter answer that is wrong scores zero.</div>
</div>
Expand Down
42 changes: 42 additions & 0 deletions evals/RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -467,3 +467,45 @@ out of 5, Haiku `csv-export` 4 out of 5, Haiku `explain-bug` 5 out of 5.

About 26 USD in all. Three Sonnet runs did not start in the first suite
(the same process spawn error) and their two cases were run again.

## The table on six runs (crewcut 0.17.0, 2026-11-13)

The previous section showed that a suite of three runs does not separate
0.97 from 1.00. A second full suite was run, both arms, four models, and
every run measured with the 0.17.0 wording and graders is pooled: six runs
per case, nine with the plugin on Sonnet and Haiku, which had been run
twice already.

| Model | Score with | without | Runs with | without | Turns with | without | Session cost with | without |
|---|--:|--:|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 1.00 | 0.87 | 54 | 54 | 4.7 | 5.5 | 0.203 USD | 0.189 USD |
| Opus 5.5 | 0.99 | 0.85 | 54 | 54 | 5.0 | 5.4 | 0.093 USD | 0.088 USD |
| Sonnet 5.5 | 0.98 | 0.87 | 81 | 54 | 4.4 | 5.3 | 0.044 USD | 0.041 USD |
| Haiku 5.5 | 0.99 | 0.84 | 81 | 54 | 5.2 | 6.0 | 0.003 USD | 0.003 USD |

Runs where every grader passes, with the plugin against without: Fable 53
out of 54 against 19 out of 54, Opus 52 against 11, Sonnet 65 out of 81
against 15 out of 54, Haiku 73 out of 81 against 8 out of 54.

Every grader that failed with the plugin, over all those runs:

- Fable: `csv-export` `reads-at-most-three-files` once.
- Opus: `csv-export` `reads-at-most-three-files` once, `explain-bug`
`full-explanation` once.
- Sonnet: `explain-bug` `full-explanation` 4 times out of 9,
`url-parse` `short-answer` 3, `date-picker` `only-asked-attributes`
3 and `short-answer` 2, `explain-retry` `comments-say-why` 2 and
`plain-text-answer` 1, `keep-validation` `honest-answer` 1,
`csv-export` `reads-at-most-three-files` 1.
- Haiku: `csv-export` without the money helper 3 times out of 9,
`date-picker` `short-answer` 2, `explain-bug` `full-explanation` 2,
`explain-retry` `comments-say-why` 1.

Sonnet is the model with the most left to gain: its two frequent misses
are an explanation that skips the caller on `explain-bug` and a fourth or
fifth sentence on `url-parse`.

The arm without the plugin moved by one point on three models between the
two suites, which is the same spread. 27.15 USD for the second suite. Five
Haiku runs of `keep-validation` did not start (the process spawn error)
and the case was run again in both arms.
10 changes: 5 additions & 5 deletions skills/crewcut-gain/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ crewcut gain measured on public repos and eval cases, not on t
Next.js boilerplate, Opus 5.5 18 % 45 % 40 % 33 % 21/21
controls on Haiku, LOC: caveman 104 % and 85 %, yagni prompt 112 % and 71 %

eval cases (nine, three runs each, same model with and without)
eval cases (nine, at least six runs each, same model with and without)

score with / without turns with / without session cost with / without
Fable 5.2 1.00 / 0.88 4.6 / 5.7 -19 % 0.203 / 0.192 USD +6 %
Opus 5.5 1.00 / 0.84 4.8 / 5.4 -11 % 0.092 / 0.087 USD +6 %
Sonnet 5.5 0.97 / 0.86 4.3 / 5.3 -19 % 0.043 / 0.041 USD +5 %
Haiku 5.5 0.97 / 0.84 5.2 / 6.0 -13 % 0.003 / 0.003 USD 0 %
Fable 5.2 1.00 / 0.87 4.7 / 5.5 -15 % 0.203 / 0.189 USD +8 %
Opus 5.5 0.99 / 0.85 5.0 / 5.4 -7 % 0.093 / 0.088 USD +6 %
Sonnet 5.5 0.98 / 0.87 4.4 / 5.3 -17 % 0.044 / 0.041 USD +7 %
Haiku 5.5 0.99 / 0.84 5.2 / 6.0 -13 % 0.003 / 0.003 USD 0 %

This repo has no baseline: the version you did not build was never written,
so no per-repo saving can be honestly computed.
Expand Down
Loading