Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,22 @@ follows Keep a Changelog and the project follows Semantic Versioning.

## [Unreleased]

### Added

- A ninth eval case, `explain-retry`: a long explanation and comments to
write, where a model reaches for long dashes, arrows and comment blocks.
Without the plugin no model writes a plain answer or one-line comments,
0 run out of 12 each. With it, 10 and 11 out of 12, and the explanation
stays complete in every run. Two weak spots: Sonnet 5.5 keeps a long
dash in 2 answers out of 3, and one comment per file often names the
mechanism instead of the reason.

### Changed

- The eval tables and the eval card count nine cases: Fable 5.2 0.98
against 0.88 without the plugin, Opus 5.5 0.99 against 0.85, Sonnet 5.5
0.98 against 0.86, Haiku 5.5 0.97 against 0.84.

## [0.15.7] - 2026-11-13

### Changed
Expand Down
12 changes: 6 additions & 6 deletions README.fr.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Le ruleset complet est dans [Les règles](#les-regles), sous Référence.

## Mesuré

<p align="center"><img src="assets/readme/evals-fr.png" width="880" alt="Suite d'évals, huit cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,98 avec crewcut, 0,89 pour le même modèle sans plugin. Opus 5.5 : 0,99 avec, 0,86 sans. Sonnet 5.5 : 0,99 avec, 0,87 sans. Haiku 5.5 : 0,98 avec, 0,86 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro."></p>
<p align="center"><img src="assets/readme/evals-fr.png" width="880" alt="Suite d'évals, neuf cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,98 avec crewcut, 0,88 pour le même modèle sans plugin. Opus 5.5 : 0,99 avec, 0,85 sans. Sonnet 5.5 : 0,98 avec, 0,86 sans. Haiku 5.5 : 0,97 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro."></p>

<p align="center"><img src="assets/readme/pt5-fr.png" width="880" alt="Le benchmark de ponytail 5, crewcut 0.14.0 face à sans plugin, Opus 5.5, 39 tâches, 5 runs. Lignes de code moins 58 pour cent, intervalle à 95 pour cent de moins 68 à moins 47, mieux sur 32 tâches et pire sur 1. Information de code moins 52 pour cent. Tokens de sortie moins 56 pour cent. Coût moins 34 pour cent, intervalle de moins 44 à moins 24. Temps moins 53 pour cent. Tours moins 19 pour cent. Vérifications cachées : 85 sur 90 réussies, 86 sans le plugin. Six tâches de sécurité : 30 sur 30 dans les deux bras. Tâche de confidentialité : 0 sur 5, puis 8 sur 10 après la règle qui nomme ce qu'un changement peut exposer. Ponytail 5 a publié pour lui-même : code moins 53 pour cent, tokens de sortie moins 45, coût moins 26, temps moins 41. Deux runs à un mois d'écart, les intervalles se recouvrent."></p>

Expand All @@ -66,15 +66,15 @@ par tâche, limites et reproduction :
<details>
<summary>Les chiffres en texte</summary>

Suite d'évals, huit cas, trois runs chacun, Opus 5.5 comme juge. Le coût est
Suite d'évals, neuf cas, trois runs chacun, Opus 5.5 comme juge. Le coût est
celui de la session seule, sans le juge :

| Modèle | crewcut | Score avec | sans | Tours avec | sans | Coût par run avec | sans |
|---|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.15.6 | 0,98 | 0,89 | 4,3 | 6,0 | 0,197 USD | 0,189 USD |
| Opus 5.5 | 0.15.6 | 0,99 | 0,86 | 4,8 | 5,7 | 0,089 USD | 0,086 USD |
| Sonnet 5.5 | 0.15.6 | 0,99 | 0,87 | 4,4 | 5,6 | 0,043 USD | 0,040 USD |
| Haiku 5.5 | 0.15.6 | 0,98 | 0,86 | 5,3 | 6,4 | 0,003 USD | 0,003 USD |
| Fable 5.2 | 0.15.7 | 0,98 | 0,88 | 4,3 | 5,7 | 0,203 USD | 0,192 USD |
| Opus 5.5 | 0.15.7 | 0,99 | 0,85 | 4,7 | 5,4 | 0,090 USD | 0,087 USD |
| Sonnet 5.5 | 0.15.7 | 0,98 | 0,86 | 4,4 | 5,3 | 0,044 USD | 0,041 USD |
| Haiku 5.5 | 0.15.7 | 0,97 | 0,84 | 5,2 | 6,0 | 0,003 USD | 0,003 USD |

Le benchmark de ponytail 5, Opus 5.5, 39 tâches, 5 runs, crewcut 0.14.0
face à sans plugin :
Expand Down
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ The full ruleset is in [The rules](#the-rules), under Reference.

## Measured

<p align="center"><img src="assets/readme/evals.png" width="880" alt="Eval suite, eight cases, three runs each. Shorter, and more often right. Fable 5.2: 0.98 with crewcut, 0.89 for the same model with no plugin. Opus 5.5: 0.99 with, 0.86 without. Sonnet 5.5: 0.99 with, 0.87 without. Haiku 5.5: 0.98 with, 0.86 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero."></p>
<p align="center"><img src="assets/readme/evals.png" width="880" alt="Eval suite, nine cases, three runs each. Shorter, and more often right. Fable 5.2: 0.98 with crewcut, 0.88 for the same model with no plugin. Opus 5.5: 0.99 with, 0.85 without. Sonnet 5.5: 0.98 with, 0.86 without. Haiku 5.5: 0.97 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero."></p>

<p align="center"><img src="assets/readme/pt5.png" width="880" alt="Ponytail 5's benchmark, crewcut 0.14.0 against no plugin, Opus 5.5, 39 tasks, 5 runs. Lines of code minus 58 percent, 95 percent interval minus 68 to minus 47, better on 32 tasks and worse on 1. Code information minus 52 percent. Output tokens minus 56 percent. Cost minus 34 percent, interval minus 44 to minus 24. Time minus 53 percent. Turns minus 19 percent. Hidden checks: 85 of 90 passed, 86 without the plugin. Six security tasks: 30 of 30 in both arms. Privacy task: 0 of 5, then 8 of 10 after the rule that names what a change could expose. Ponytail 5 published for itself: code minus 53 percent, output tokens minus 45, cost minus 26, time minus 41. Two runs a month apart, intervals overlap."></p>

Expand All @@ -66,15 +66,15 @@ limits and how to reproduce:
<details>
<summary>The figures as text</summary>

Eval suite, eight cases, three runs each, Opus 5.5 as judge. The cost is the
Eval suite, nine cases, three runs each, Opus 5.5 as judge. The cost is the
session alone, without the judge:

| Model | crewcut | Score with | without | Turns with | without | Cost per run with | without |
|---|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.15.6 | 0.98 | 0.89 | 4.3 | 6.0 | 0.197 USD | 0.189 USD |
| Opus 5.5 | 0.15.6 | 0.99 | 0.86 | 4.8 | 5.7 | 0.089 USD | 0.086 USD |
| Sonnet 5.5 | 0.15.6 | 0.99 | 0.87 | 4.4 | 5.6 | 0.043 USD | 0.040 USD |
| Haiku 5.5 | 0.15.6 | 0.98 | 0.86 | 5.3 | 6.4 | 0.003 USD | 0.003 USD |
| Fable 5.2 | 0.15.7 | 0.98 | 0.88 | 4.3 | 5.7 | 0.203 USD | 0.192 USD |
| Opus 5.5 | 0.15.7 | 0.99 | 0.85 | 4.7 | 5.4 | 0.090 USD | 0.087 USD |
| Sonnet 5.5 | 0.15.7 | 0.98 | 0.86 | 4.4 | 5.3 | 0.044 USD | 0.041 USD |
| Haiku 5.5 | 0.15.7 | 0.97 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD |

Ponytail 5's benchmark, Opus 5.5, 39 tasks, 5 runs, crewcut 0.14.0 against
no plugin:
Expand Down
Binary file modified assets/readme/evals-fr.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/readme/evals.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 5 additions & 5 deletions assets/readme/src/evals-fr.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@
.legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
</style>
<div class="card"><div class="pole"></div>
<div class="eyebrow">Suite d'évals, huit cas, trois runs chacun</div>
<div class="eyebrow">Suite d'évals, neuf cas, trois runs chacun</div>
<h1>Plus court, et <em>plus souvent juste.</em></h1>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:89%">0,89</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:99%">0,99</div><div class="bar o" style="width:86%">0,86</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:99%">0,99</div><div class="bar o" style="width:87%">0,87</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:86%">0,86</div></div></div>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:88%">0,88</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:99%">0,99</div><div class="bar o" style="width:85%">0,85</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:86%">0,86</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:97%">0,97</div><div class="bar o" style="width:84%">0,84</div></div></div>
<div class="legend"><span><i style="background:#E34948"></i>avec crewcut</span><span><i style="background:#2A3945"></i>même modèle, sans plugin</span></div>
<div class="foot">Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille :<br>une réponse plus courte mais fausse vaut zéro.</div>
</div>
Expand Down
10 changes: 5 additions & 5 deletions assets/readme/src/evals.html
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@
.legend i { display: inline-block; width: 12px; height: 12px; border-radius: 3px; margin-right: 8px; }
</style>
<div class="card"><div class="pole"></div>
<div class="eyebrow">Eval suite, eight cases, three runs each</div>
<div class="eyebrow">Eval suite, nine cases, three runs each</div>
<h1>Shorter, and <em>more often right.</em></h1>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:89%">0.89</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:99%">0.99</div><div class="bar o" style="width:86%">0.86</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:99%">0.99</div><div class="bar o" style="width:87%">0.87</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:86%">0.86</div></div></div>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:88%">0.88</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:99%">0.99</div><div class="bar o" style="width:85%">0.85</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:86%">0.86</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:97%">0.97</div><div class="bar o" style="width:84%">0.84</div></div></div>
<div class="legend"><span><i style="background:#E34948"></i>with crewcut</span><span><i style="background:#2A3945"></i>same model, no plugin</span></div>
<div class="foot">Score is the share of graders passed. Every case grades correctness as well as size:<br>a shorter answer that is wrong scores zero.</div>
</div>
Expand Down
53 changes: 53 additions & 0 deletions evals/RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -304,3 +304,56 @@ boundary and fails if any check disappears; `explain-bug` asks a question
and fails if the explanation is cut short; `vat-country` and `csv-export`
fail if an existing test or an untouched module changes.

## A ninth case for long text and comments: explain-retry (crewcut 0.15.7, 2026-11-13)

On the older cases the plain text and comment graders passed almost
everywhere, in both arms: short tasks do not invite long prose or comment
blocks. This case does. One file, a fetch with retries and no comment; the
ticket: "A new hire starts on Monday. Explain in your answer how
fetchWithRetry in src/retry.js works, step by step, and add the comments
the code needs." Fourteen graders: the code is untouched, the answer covers
six behaviours (missing two fails), no banned character in the answer or
the file, no block comment and no two comment lines in a row, every comment
gives a reason.

| Runs passed, with / without the plugin | Fable | Opus | Sonnet | Haiku |
|---|--:|--:|--:|--:|
| Score | 1.00 / 0.79 | 0.94 / 0.70 | 0.88 / 0.73 | 0.88 / 0.70 |
| full-explanation | 3 / 3 | 3 / 3 | 3 / 3 | 3 / 3 |
| plain-text-answer | 3 / 0 | 3 / 0 | 1 / 0 | 3 / 0 |
| comments-one-line | 3 / 0 | 3 / 0 | 3 / 0 | 2 / 0 |
| comments-say-why | 3 / 2 | 1 / 0 | 1 / 0 | 0 / 0 |

Three runs per arm, 4.46 USD with the judge. The explanation is complete in
every run of both arms: the plugin's shorter answer loses nothing here.

**Without the plugin, no model writes a plain answer and none keeps its
comments to one line**: 0 run out of 12 on each. An explanation in steps
brings long dashes and arrows, and "add the comments" brings blocks.

**With the plugin, two weak spots.** Sonnet still puts a long dash in its
answer in 2 runs out of 3: the plain text rule holds on short answers and
gives way on a long one. And `comments-say-why` passes in 5 runs out of 12
with the plugin. The comments were read: most give a reason ("4xx is the
caller's fault, so retrying cannot help"), and one per file names the
mechanism instead ("Exponential backoff, capped at 5s"). The grader asks
that every comment give a reason, so one such line fails the file. That is
what "why only" says; the rule is not yet followed to the letter.

A first launch of this run found no `claude` command for any model: the
Claude Code binary was being rewritten on disk at that minute, at the same
version, as during the benchmark earlier in the day. No session had
started; the run was launched again.

## Nine cases (crewcut 0.15.7, 2026-11-13)

| Model | Score with | without | Turns with | without | Session cost with | without |
|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.98 | 0.88 | 4.3 | 5.7 | 0.203 USD | 0.192 USD |
| Opus 5.5 | 0.99 | 0.85 | 4.7 | 5.4 | 0.090 USD | 0.087 USD |
| Sonnet 5.5 | 0.98 | 0.86 | 4.4 | 5.3 | 0.044 USD | 0.041 USD |
| Haiku 5.5 | 0.97 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD |

As for the eight-case table, this joins measures of the same day on the
same plugin code: the eight cases and `explain-retry`. 54 runs per model,
none in error.
73 changes: 73 additions & 0 deletions evals/explain-retry/case.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
schema_version: "1.0"
name: explain-retry
description: "A request for a long explanation and for comments, where a model reaches for arrows, long dashes, emoji and comment blocks. Expected: a full explanation in plain text, few one-line comments that give a reason, the code untouched."
context:
scaffold_script: scaffold.sh
execution:
prompt: A new hire starts on Monday. Explain in your answer how fetchWithRetry in src/retry.js works, step by step, and add the comments the code needs.
max_turns: 10
timeout_seconds: 300
allowed_tools: [Read, Edit, Grep, Glob, Skill]
runs: 3
graders:
- type: file_exists
name: no-new-file
path: "**/*"
exists: false
- type: regex
name: keeps-attempts
target: { source: file, path: src/retry.js }
pattern: 'attempts = 4'
match: contains
- type: regex
name: keeps-status-check
target: { source: file, path: src/retry.js }
pattern: 'if \(response\.status < 500\) return response;'
match: contains
- type: regex
name: keeps-abort
target: { source: file, path: src/retry.js }
pattern: 'if \(error\.name === ''AbortError''\) throw error;'
match: contains
- type: regex
name: keeps-backoff
target: { source: file, path: src/retry.js }
pattern: 'delay = Math\.min\(delay \* 2, 5000\);'
match: contains
- type: regex
name: plain-text-answer
pattern: '[\u2013\u2014\u2018\u2019\u201C\u201D\u00A0\u202F\u2190-\u21FF\u2600-\u27BF\uD83C-\uD83E]'
match: not_contains
- type: regex
name: plain-text-in-code
target: { source: file, path: src/retry.js }
pattern: '[\u2013\u2014\u2018\u2019\u201C\u201D\u00A0\u202F\u2190-\u21FF\u2600-\u27BF\uD83C-\uD83E]'
match: not_contains
- type: regex
name: no-crewcut-in-code
target: { source: file, path: src/retry.js }
pattern: '[Cc]rewcut'
match: not_contains
- type: regex
name: comments-one-line
target: { source: file, path: src/retry.js }
pattern: '/\*|//[^\n]*\n\s*//'
match: not_contains
- type: llm
name: comments-say-why
focus: { source: file, path: src/retry.js }
criteria: >
Every comment in the file gives a reason or a constraint the code
cannot show. No comment restates what the next line does, and none
names a tool or an assistant. A file with no comment fails: the
request asked for the comments the code needs.
- type: llm
name: full-explanation
criteria: >
The final message explains all of these: it tries up to four times by
default; a response with a status below 500, 4xx included, is returned
at once and not retried; a 5xx response or a network error is retried;
an aborted request is rethrown at once; the wait starts at 200 ms,
doubles each time, is capped at 5 seconds and gets a small random
extra; after the last attempt the last error is thrown. Missing two or
more of these fails.
33 changes: 33 additions & 0 deletions evals/explain-retry/scaffold.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
#!/usr/bin/env bash
set -eu
cat > package.json <<'EOF'
{
"name": "demo-http-client",
"version": "1.0.0",
"private": true,
"type": "module",
"engines": { "node": ">=22" }
}
EOF
mkdir -p src
cat > src/retry.js <<'EOF'
export async function fetchWithRetry(url, options = {}, attempts = 4) {
let delay = 200;
let lastError;
for (let attempt = 1; attempt <= attempts; attempt++) {
try {
const response = await fetch(url, options);
if (response.status < 500) return response;
lastError = new Error(`HTTP ${response.status}`);
} catch (error) {
if (error.name === 'AbortError') throw error;
lastError = error;
}
if (attempt < attempts) {
await new Promise((resolve) => setTimeout(resolve, delay + Math.random() * 100));
delay = Math.min(delay * 2, 5000);
}
}
throw lastError;
}
EOF
10 changes: 5 additions & 5 deletions skills/crewcut-gain/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ crewcut gain measured on public repos and eval cases, not on t
Next.js boilerplate, Opus 5.5 18 % 45 % 40 % 33 % 21/21
controls on Haiku, LOC: caveman 104 % and 85 %, yagni prompt 112 % and 71 %

eval cases (eight, three runs each, same model with and without)
eval cases (nine, three runs each, same model with and without)

score with / without turns with / without session cost with / without
Fable 5.2 0.98 / 0.89 4.3 / 6.0 -28 % 0.197 / 0.189 USD +4 %
Opus 5.5 0.99 / 0.86 4.8 / 5.7 -16 % 0.089 / 0.086 USD +3 %
Sonnet 5.5 0.99 / 0.87 4.4 / 5.6 -21 % 0.043 / 0.040 USD +8 %
Haiku 5.5 0.98 / 0.86 5.3 / 6.4 -17 % 0.003 / 0.003 USD 0 %
Fable 5.2 0.98 / 0.88 4.3 / 5.7 -25 % 0.203 / 0.192 USD +6 %
Opus 5.5 0.99 / 0.85 4.7 / 5.4 -13 % 0.090 / 0.087 USD +3 %
Sonnet 5.5 0.98 / 0.86 4.4 / 5.3 -17 % 0.044 / 0.041 USD +7 %
Haiku 5.5 0.97 / 0.84 5.2 / 6.0 -13 % 0.003 / 0.003 USD 0 %

This repo has no baseline: the version you did not build was never written,
so no per-repo saving can be honestly computed.
Expand Down
Loading