Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Comments: only a reason the code cannot show, one plain line; never a summary of
Modern by default: follow the installed versions' current docs, compact forms when clear; unsure: read them; never a feature the version lacks.

Tokens:
- Output: no preamble, recap or unrequested explanation; never paste back your code. Three sentences at most: what changed, where, one caveat only if it changes what the user does next; no offer or skipped line for what the ticket never asked; no bullets.
- Output: no preamble, recap or unrequested explanation; never paste back your code. Three sentences at most: what changed (a bug: its cause first), where, one caveat only if it changes what the user does next; no offer or skipped line for what the ticket never asked; no bullets.
- Reading: grep what the change touches, then read only those files by line range; never read a file twice or to confirm what grep showed; no repo tour.
- Writing: targeted edits, no whole-file rewrite, no unrequested docs or refactors; one test run.
- Tests: none unless the task asks or an existing test file covers the touched code, then extend it; never create one, even if told to add tests "if you normally would".
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,21 @@ follows Keep a Changelog and the project follows Semantic Versioning.

## [Unreleased]

### Changed

- The answer to a bug fix names the cause first. The Output rule now reads
"what changed (a bug: its cause first), where". Fable 5.2 states the
cause on `shared-bug` in 5 runs out of 5, against 1 out of 3, and Opus
5.5 on `csv-export` in 5 out of 5, against 2 out of 3.
- The ruleset ceiling goes from 630 to 640 estimated tokens: the clause
takes it to 634 at its heaviest level.
- Eval suite: the `short-answer` grader of `date-picker` accepts an
answer that says the field was made required and can be made optional.
- Eval suite, nine cases: Fable 5.2 1.00 against 0.88 without the plugin,
Opus 5.5 1.00 against 0.84, Sonnet 5.5 0.97 against 0.86, Haiku 5.5 0.97
against 0.84. Sonnet and Haiku move between 0.97 and 1.00 from one suite
to the next under either wording; the notebook has both.

## [0.16.0] - 2026-11-13

### Changed
Expand Down
12 changes: 6 additions & 6 deletions README.fr.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Le ruleset complet est dans [Les règles](#les-regles), sous Référence.

## Mesuré

<p align="center"><img src="assets/readme/evals-fr.png" width="880" alt="Suite d'évals, neuf cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,99 avec crewcut, 0,88 pour le même modèle sans plugin. Opus 5.5 : 1,00 avec, 0,85 sans. Sonnet 5.5 : 1,00 avec, 0,86 sans. Haiku 5.5 : 0,98 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro."></p>
<p align="center"><img src="assets/readme/evals-fr.png" width="880" alt="Suite d'évals, neuf cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 1,00 avec crewcut, 0,88 pour le même modèle sans plugin. Opus 5.5 : 1,00 avec, 0,84 sans. Sonnet 5.5 : 0,97 avec, 0,86 sans. Haiku 5.5 : 0,97 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro."></p>

<p align="center"><img src="assets/readme/pt5-fr.png" width="880" alt="Le benchmark de ponytail 5, crewcut 0.14.0 face à sans plugin, Opus 5.5, 39 tâches, 5 runs. Lignes de code moins 58 pour cent, intervalle à 95 pour cent de moins 68 à moins 47, mieux sur 32 tâches et pire sur 1. Information de code moins 52 pour cent. Tokens de sortie moins 56 pour cent. Coût moins 34 pour cent, intervalle de moins 44 à moins 24. Temps moins 53 pour cent. Tours moins 19 pour cent. Vérifications cachées : 85 sur 90 réussies, 86 sans le plugin. Six tâches de sécurité : 30 sur 30 dans les deux bras. Tâche de confidentialité : 0 sur 5, puis 8 sur 10 après la règle qui nomme ce qu'un changement peut exposer. Ponytail 5 a publié pour lui-même : code moins 53 pour cent, tokens de sortie moins 45, coût moins 26, temps moins 41. Deux runs à un mois d'écart, les intervalles se recouvrent."></p>

Expand All @@ -71,10 +71,10 @@ celui de la session seule, sans le juge :

| Modèle | crewcut | Score avec | sans | Tours avec | sans | Coût par run avec | sans |
|---|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.16.0 | 0,99 | 0,88 | 4,6 | 5,7 | 0,201 USD | 0,192 USD |
| Opus 5.5 | 0.16.0 | 1,00 | 0,85 | 5,0 | 5,4 | 0,093 USD | 0,087 USD |
| Sonnet 5.5 | 0.16.0 | 1,00 | 0,86 | 4,5 | 5,3 | 0,044 USD | 0,041 USD |
| Haiku 5.5 | 0.16.0 | 0,98 | 0,84 | 5,1 | 6,0 | 0,003 USD | 0,003 USD |
| Fable 5.2 | 0.17.0 | 1,00 | 0,88 | 4,6 | 5,7 | 0,203 USD | 0,192 USD |
| Opus 5.5 | 0.17.0 | 1,00 | 0,84 | 4,8 | 5,4 | 0,092 USD | 0,087 USD |
| Sonnet 5.5 | 0.17.0 | 0,97 | 0,86 | 4,3 | 5,3 | 0,043 USD | 0,041 USD |
| Haiku 5.5 | 0.17.0 | 0,97 | 0,84 | 5,2 | 6,0 | 0,003 USD | 0,003 USD |

Le benchmark de ponytail 5, Opus 5.5, 39 tâches, 5 runs, crewcut 0.14.0
face à sans plugin :
Expand Down Expand Up @@ -252,7 +252,7 @@ Ce que dit le ruleset, au-delà de l'échelle plus haut.

- Sortie : pas de préambule, pas de récapitulatif, pas d'explication non
demandée. Ne jamais recoller le code qui vient d'être écrit. Trois phrases
au plus : ce qui a changé, où, et une mise en garde seulement si elle change
au plus : ce qui a changé, pour un bug sa cause d'abord, où, et une mise en garde seulement si elle change
ce que vous ferez ensuite ; aucune proposition ni ligne "skipped" pour ce
que le ticket n'a pas demandé ; pas de liste à puces.
- Lecture : grep sur les symboles que le changement touche, puis lire
Expand Down
13 changes: 7 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ The full ruleset is in [The rules](#the-rules), under Reference.

## Measured

<p align="center"><img src="assets/readme/evals.png" width="880" alt="Eval suite, nine cases, three runs each. Shorter, and more often right. Fable 5.2: 0.99 with crewcut, 0.88 for the same model with no plugin. Opus 5.5: 1.00 with, 0.85 without. Sonnet 5.5: 1.00 with, 0.86 without. Haiku 5.5: 0.98 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero."></p>
<p align="center"><img src="assets/readme/evals.png" width="880" alt="Eval suite, nine cases, three runs each. Shorter, and more often right. Fable 5.2: 1.00 with crewcut, 0.88 for the same model with no plugin. Opus 5.5: 1.00 with, 0.84 without. Sonnet 5.5: 0.97 with, 0.86 without. Haiku 5.5: 0.97 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero."></p>

<p align="center"><img src="assets/readme/pt5.png" width="880" alt="Ponytail 5's benchmark, crewcut 0.14.0 against no plugin, Opus 5.5, 39 tasks, 5 runs. Lines of code minus 58 percent, 95 percent interval minus 68 to minus 47, better on 32 tasks and worse on 1. Code information minus 52 percent. Output tokens minus 56 percent. Cost minus 34 percent, interval minus 44 to minus 24. Time minus 53 percent. Turns minus 19 percent. Hidden checks: 85 of 90 passed, 86 without the plugin. Six security tasks: 30 of 30 in both arms. Privacy task: 0 of 5, then 8 of 10 after the rule that names what a change could expose. Ponytail 5 published for itself: code minus 53 percent, output tokens minus 45, cost minus 26, time minus 41. Two runs a month apart, intervals overlap."></p>

Expand All @@ -71,10 +71,10 @@ session alone, without the judge:

| Model | crewcut | Score with | without | Turns with | without | Cost per run with | without |
|---|---|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 0.16.0 | 0.99 | 0.88 | 4.6 | 5.7 | 0.201 USD | 0.192 USD |
| Opus 5.5 | 0.16.0 | 1.00 | 0.85 | 5.0 | 5.4 | 0.093 USD | 0.087 USD |
| Sonnet 5.5 | 0.16.0 | 1.00 | 0.86 | 4.5 | 5.3 | 0.044 USD | 0.041 USD |
| Haiku 5.5 | 0.16.0 | 0.98 | 0.84 | 5.1 | 6.0 | 0.003 USD | 0.003 USD |
| Fable 5.2 | 0.17.0 | 1.00 | 0.88 | 4.6 | 5.7 | 0.203 USD | 0.192 USD |
| Opus 5.5 | 0.17.0 | 1.00 | 0.84 | 4.8 | 5.4 | 0.092 USD | 0.087 USD |
| Sonnet 5.5 | 0.17.0 | 0.97 | 0.86 | 4.3 | 5.3 | 0.043 USD | 0.041 USD |
| Haiku 5.5 | 0.17.0 | 0.97 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD |

Ponytail 5's benchmark, Opus 5.5, 39 tasks, 5 runs, crewcut 0.14.0 against
no plugin:
Expand Down Expand Up @@ -247,7 +247,8 @@ What the ruleset says, beyond the ladder above.
**Token discipline**

- Output: no preamble, no recap, no unrequested explanation. Never paste
back code just written. Three sentences at most: what changed, where, and
back code just written. Three sentences at most: what changed, for a bug its
cause first, where, and
one caveat only if it changes what you do next; no offer and no "skipped"
line for what the ticket never asked; no bullet list.
- Reading: grep for the symbols the change touches, then read only those
Expand Down
Binary file modified assets/readme/evals-fr.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified assets/readme/evals.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 4 additions & 4 deletions assets/readme/src/evals-fr.html
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@
<div class="card"><div class="pole"></div>
<div class="eyebrow">Suite d'évals, neuf cas, trois runs chacun</div>
<h1>Plus court, et <em>plus souvent juste.</em></h1>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:99%">0,99</div><div class="bar o" style="width:88%">0,88</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:100%">1,00</div><div class="bar o" style="width:85%">0,85</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:100%">1,00</div><div class="bar o" style="width:86%">0,86</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:98%">0,98</div><div class="bar o" style="width:84%">0,84</div></div></div>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:100%">1,00</div><div class="bar o" style="width:88%">0,88</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:100%">1,00</div><div class="bar o" style="width:84%">0,84</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:97%">0,97</div><div class="bar o" style="width:86%">0,86</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:97%">0,97</div><div class="bar o" style="width:84%">0,84</div></div></div>
<div class="legend"><span><i style="background:#E34948"></i>avec crewcut</span><span><i style="background:#2A3945"></i>même modèle, sans plugin</span></div>
<div class="foot">Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille :<br>une réponse plus courte mais fausse vaut zéro.</div>
</div>
Expand Down
8 changes: 4 additions & 4 deletions assets/readme/src/evals.html
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,10 @@
<div class="card"><div class="pole"></div>
<div class="eyebrow">Eval suite, nine cases, three runs each</div>
<h1>Shorter, and <em>more often right.</em></h1>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:99%">0.99</div><div class="bar o" style="width:88%">0.88</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:100%">1.00</div><div class="bar o" style="width:85%">0.85</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:100%">1.00</div><div class="bar o" style="width:86%">0.86</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:98%">0.98</div><div class="bar o" style="width:84%">0.84</div></div></div>
<div class="row"><div class="m">Fable 5.2</div><div><div class="bar w" style="width:100%">1.00</div><div class="bar o" style="width:88%">0.88</div></div></div>
<div class="row"><div class="m">Opus 5.5</div><div><div class="bar w" style="width:100%">1.00</div><div class="bar o" style="width:84%">0.84</div></div></div>
<div class="row"><div class="m">Sonnet 5.5</div><div><div class="bar w" style="width:97%">0.97</div><div class="bar o" style="width:86%">0.86</div></div></div>
<div class="row"><div class="m">Haiku 5.5</div><div><div class="bar w" style="width:97%">0.97</div><div class="bar o" style="width:84%">0.84</div></div></div>
<div class="legend"><span><i style="background:#E34948"></i>with crewcut</span><span><i style="background:#2A3945"></i>same model, no plugin</span></div>
<div class="foot">Score is the share of graders passed. Every case grades correctness as well as size:<br>a shorter answer that is wrong scores zero.</div>
</div>
Expand Down
62 changes: 62 additions & 0 deletions evals/RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -405,3 +405,65 @@ twice; Opus, `csv-export` `root-cause-stated` once; Haiku, five graders
once each. 12.42 USD for this table, 7.09 USD for the three trials. Six
Fable runs did not start (a process spawn error, twice that day already)
and their two cases were run again.

## The cause of a bug in the answer (crewcut 0.17.0, 2026-11-13)

Three graders kept failing with the plugin on. Read side by side, the
failing and the passing answers gave two different stories.

On `shared-bug` and `csv-export`, `root-cause-stated` failed when the
answer described the fix and left the cause to be inferred: "formatPrice
now converts numeric strings to numbers". The ruleset asked for "what
changed, where" and nothing more, and Fable, the model that holds the
three sentences most strictly, spent them on the fix and two caveats. The
Output line now reads "what changed (a bug: its cause first), where".

| Plugin arm, `root-cause-stated` | Before | After |
|---|--:|--:|
| Fable 5.2, `shared-bug` | 1 / 3 | 5 / 5 |
| Opus 5.5, `csv-export` | 2 / 3 | 5 / 5 |

On `date-picker`, every `short-answer` failure carried the same
sentence: the field was made required like the others, "drop required if
it should be optional". The passing answers made the same choice and said
it as a fact, or did not say it. The grader was written against offers of
a script or of a computed max date; naming a choice the ticket left open
and how to undo it is not one, so the grader now allows it. This is a
change of the measure, not of the plugin, and the arm without the plugin
was run again on that case with the new grader.

The clause takes the ruleset to 634 estimated tokens at its heaviest
level; the ceiling goes from 630 to 640.

The nine cases with the plugin, three runs, Opus as judge. The arm without
the plugin is the one of the earlier sections, `date-picker` excepted.

| Model | Score with | was | without | Turns with | without | Session cost with | without |
|---|--:|--:|--:|--:|--:|--:|--:|
| Fable 5.2 | 1.00 | 0.99 | 0.88 | 4.6 | 5.7 | 0.203 USD | 0.192 USD |
| Opus 5.5 | 1.00 | 1.00 | 0.84 | 4.8 | 5.4 | 0.092 USD | 0.087 USD |
| Sonnet 5.5 | 0.97 | 1.00 | 0.86 | 4.3 | 5.3 | 0.043 USD | 0.041 USD |
| Haiku 5.5 | 0.97 | 0.98 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD |

Fable and Opus pass every grader of every run. Sonnet and Haiku came out
lower than in the previous section, so both were run a second time, with
the new wording and with the one of 0.16.0:

| Score with the plugin, nine cases, three runs | 0.16.0 wording | 0.17.0 wording |
|---|--:|--:|
| Sonnet 5.5, first suite | 1.00 | 0.97 |
| Sonnet 5.5, second suite | 0.97 | 0.98 |
| Haiku 5.5, first suite | 0.98 | 0.97 |
| Haiku 5.5, second suite | 0.99 | 0.99 |

The same graders fail under both wordings: on Sonnet `explain-bug`
`full-explanation` and `url-parse` `short-answer`, twice each in a
suite, when the session reads one file less. The spread between two suites
of the same wording is as wide as the gap between the wordings: a suite of
three runs does not separate 0.97 from 1.00 on these two models, and the
1.00 published for Sonnet in 0.16.0 was the high end of that spread. Three
cases checked on five runs with the new wording: Sonnet `explain-bug` 4
out of 5, Haiku `csv-export` 4 out of 5, Haiku `explain-bug` 5 out of 5.

About 26 USD in all. Three Sonnet runs did not start in the first suite
(the same process spawn error) and their two cases were run again.
3 changes: 2 additions & 1 deletion evals/date-picker/case.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,8 @@ graders:
bullet points. It names the file and the field. It has at most four
sentences in all. No sentence offers or suggests a script, a computed
or dynamic value, or another change to the form; saying that the server
still has to accept or check the field is fine.
still has to accept or check the field is fine, and so is saying that
the field was made required and that the attribute can be dropped.
- type: regex
name: plain-text-answer
pattern: '[\u2013\u2014\u2018\u2019\u201C\u201D\u00A0\u202F\u2190-\u21FF\u2600-\u27BF\uD83C-\uD83E]'
Expand Down
2 changes: 1 addition & 1 deletion hooks/ruleset.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Comments: only a reason the code cannot show, one plain line; never a summary of
Modern by default: follow the installed versions' current docs, compact forms when clear; unsure: read them; never a feature the version lacks.

Tokens:
- Output: no preamble, recap or unrequested explanation; never paste back your code. Three sentences at most: what changed, where, one caveat only if it changes what the user does next; no offer or skipped line for what the ticket never asked; no bullets.
- Output: no preamble, recap or unrequested explanation; never paste back your code. Three sentences at most: what changed (a bug: its cause first), where, one caveat only if it changes what the user does next; no offer or skipped line for what the ticket never asked; no bullets.
[lite] - Reading: prefer grep to reading whole files.
[full] - Reading: grep what the change touches, then read only those files by line range; never read a file twice or to confirm what grep showed; no repo tour.
[ultra] - Reading: grep first, then one read per file the change touches, by line range, never twice; no repo tour.
Expand Down
8 changes: 4 additions & 4 deletions skills/crewcut-gain/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,10 @@ crewcut gain measured on public repos and eval cases, not on t
eval cases (nine, three runs each, same model with and without)

score with / without turns with / without session cost with / without
Fable 5.2 0.99 / 0.88 4.6 / 5.7 -19 % 0.201 / 0.192 USD +5 %
Opus 5.5 1.00 / 0.85 5.0 / 5.4 -7 % 0.093 / 0.087 USD +7 %
Sonnet 5.5 1.00 / 0.86 4.5 / 5.3 -15 % 0.044 / 0.041 USD +7 %
Haiku 5.5 0.98 / 0.84 5.1 / 6.0 -15 % 0.003 / 0.003 USD 0 %
Fable 5.2 1.00 / 0.88 4.6 / 5.7 -19 % 0.203 / 0.192 USD +6 %
Opus 5.5 1.00 / 0.84 4.8 / 5.4 -11 % 0.092 / 0.087 USD +6 %
Sonnet 5.5 0.97 / 0.86 4.3 / 5.3 -19 % 0.043 / 0.041 USD +5 %
Haiku 5.5 0.97 / 0.84 5.2 / 6.0 -13 % 0.003 / 0.003 USD 0 %

This repo has no baseline: the version you did not build was never written,
so no per-repo saving can be honestly computed.
Expand Down
2 changes: 1 addition & 1 deletion skills/crewcut/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ second ticket. No type alias or helper for a single use.
## Output

Never paste back code you wrote. Reply in three sentences at most: what
changed and where, then one caveat, only if it changes what the user does
changed (for a bug, its cause first) and where, then one caveat, only if it changes what the user does
next. `skipped: <what>, add when: <condition>` only for a corner the ticket
needed and you cut; never for an option it did not ask for. No preamble,
restating or recap. An explanation the user asked for is given in full; only
Expand Down
4 changes: 2 additions & 2 deletions tests/text.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -66,12 +66,12 @@ test('commit messages use plain punctuation and carry no trailer', () => {
assert.doesNotMatch(log, /^[A-Za-z-]+: .+$/m, 'trailer line found in a commit message');
});

test('compact ruleset stays under 630 estimated tokens at every level', () => {
test('compact ruleset stays under 640 estimated tokens at every level', () => {
const markdown = fs.readFileSync(path.join(root, 'hooks', 'ruleset.md'), 'utf8');
for (const level of LEVELS) {
for (const markers of [false, true]) {
const tokens = Math.ceil(renderRuleset(level, markdown, { markers }).length / 4);
assert.ok(tokens < 630, `${level}, markers ${markers}: about ${tokens} tokens`);
assert.ok(tokens < 640, `${level}, markers ${markers}: about ${tokens} tokens`);
}
}
});
Expand Down
Loading