diff --git a/AGENTS.md b/AGENTS.md index 67d3cff..e6a8c07 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ Comments: only a reason the code cannot show, one plain line; never a summary of Modern by default: follow the installed versions' current docs, compact forms when clear; unsure: read them; never a feature the version lacks. Tokens: -- Output: no preamble, recap or unrequested explanation; never paste back your code. Three sentences at most: what changed, where, one caveat only if it changes what the user does next; no offer or skipped line for what the ticket never asked; no bullets. +- Output: no preamble, recap or unrequested explanation; never paste back your code. Three sentences at most: what changed (a bug: its cause first), where, one caveat only if it changes what the user does next; no offer or skipped line for what the ticket never asked; no bullets. - Reading: grep what the change touches, then read only those files by line range; never read a file twice or to confirm what grep showed; no repo tour. - Writing: targeted edits, no whole-file rewrite, no unrequested docs or refactors; one test run. - Tests: none unless the task asks or an existing test file covers the touched code, then extend it; never create one, even if told to add tests "if you normally would". diff --git a/CHANGELOG.md b/CHANGELOG.md index cb40fc8..119d48d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,21 @@ follows Keep a Changelog and the project follows Semantic Versioning. ## [Unreleased] +### Changed + +- The answer to a bug fix names the cause first. The Output rule now reads + "what changed (a bug: its cause first), where". Fable 5.2 states the + cause on `shared-bug` in 5 runs out of 5, against 1 out of 3, and Opus + 5.5 on `csv-export` in 5 out of 5, against 2 out of 3. +- The ruleset ceiling goes from 630 to 640 estimated tokens: the clause + takes it to 634 at its heaviest level. +- Eval suite: the `short-answer` grader of `date-picker` accepts an + answer that says the field was made required and can be made optional. +- Eval suite, nine cases: Fable 5.2 1.00 against 0.88 without the plugin, + Opus 5.5 1.00 against 0.84, Sonnet 5.5 0.97 against 0.86, Haiku 5.5 0.97 + against 0.84. Sonnet and Haiku move between 0.97 and 1.00 from one suite + to the next under either wording; the notebook has both. + ## [0.16.0] - 2026-11-13 ### Changed diff --git a/README.fr.md b/README.fr.md index 97302e3..d5e19b2 100644 --- a/README.fr.md +++ b/README.fr.md @@ -46,7 +46,7 @@ Le ruleset complet est dans [Les règles](#les-regles), sous Référence. ## Mesuré -

Suite d'évals, neuf cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 0,99 avec crewcut, 0,88 pour le même modèle sans plugin. Opus 5.5 : 1,00 avec, 0,85 sans. Sonnet 5.5 : 1,00 avec, 0,86 sans. Haiku 5.5 : 0,98 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro.

+

Suite d'évals, neuf cas, trois runs chacun. Plus court, et plus souvent juste. Fable 5.2 : 1,00 avec crewcut, 0,88 pour le même modèle sans plugin. Opus 5.5 : 1,00 avec, 0,84 sans. Sonnet 5.5 : 0,97 avec, 0,86 sans. Haiku 5.5 : 0,97 avec, 0,84 sans. Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille : une réponse plus courte mais fausse vaut zéro.

Le benchmark de ponytail 5, crewcut 0.14.0 face à sans plugin, Opus 5.5, 39 tâches, 5 runs. Lignes de code moins 58 pour cent, intervalle à 95 pour cent de moins 68 à moins 47, mieux sur 32 tâches et pire sur 1. Information de code moins 52 pour cent. Tokens de sortie moins 56 pour cent. Coût moins 34 pour cent, intervalle de moins 44 à moins 24. Temps moins 53 pour cent. Tours moins 19 pour cent. Vérifications cachées : 85 sur 90 réussies, 86 sans le plugin. Six tâches de sécurité : 30 sur 30 dans les deux bras. Tâche de confidentialité : 0 sur 5, puis 8 sur 10 après la règle qui nomme ce qu'un changement peut exposer. Ponytail 5 a publié pour lui-même : code moins 53 pour cent, tokens de sortie moins 45, coût moins 26, temps moins 41. Deux runs à un mois d'écart, les intervalles se recouvrent.

@@ -71,10 +71,10 @@ celui de la session seule, sans le juge : | Modèle | crewcut | Score avec | sans | Tours avec | sans | Coût par run avec | sans | |---|---|--:|--:|--:|--:|--:|--:| -| Fable 5.2 | 0.16.0 | 0,99 | 0,88 | 4,6 | 5,7 | 0,201 USD | 0,192 USD | -| Opus 5.5 | 0.16.0 | 1,00 | 0,85 | 5,0 | 5,4 | 0,093 USD | 0,087 USD | -| Sonnet 5.5 | 0.16.0 | 1,00 | 0,86 | 4,5 | 5,3 | 0,044 USD | 0,041 USD | -| Haiku 5.5 | 0.16.0 | 0,98 | 0,84 | 5,1 | 6,0 | 0,003 USD | 0,003 USD | +| Fable 5.2 | 0.17.0 | 1,00 | 0,88 | 4,6 | 5,7 | 0,203 USD | 0,192 USD | +| Opus 5.5 | 0.17.0 | 1,00 | 0,84 | 4,8 | 5,4 | 0,092 USD | 0,087 USD | +| Sonnet 5.5 | 0.17.0 | 0,97 | 0,86 | 4,3 | 5,3 | 0,043 USD | 0,041 USD | +| Haiku 5.5 | 0.17.0 | 0,97 | 0,84 | 5,2 | 6,0 | 0,003 USD | 0,003 USD | Le benchmark de ponytail 5, Opus 5.5, 39 tâches, 5 runs, crewcut 0.14.0 face à sans plugin : @@ -252,7 +252,7 @@ Ce que dit le ruleset, au-delà de l'échelle plus haut. - Sortie : pas de préambule, pas de récapitulatif, pas d'explication non demandée. Ne jamais recoller le code qui vient d'être écrit. Trois phrases - au plus : ce qui a changé, où, et une mise en garde seulement si elle change + au plus : ce qui a changé, pour un bug sa cause d'abord, où, et une mise en garde seulement si elle change ce que vous ferez ensuite ; aucune proposition ni ligne "skipped" pour ce que le ticket n'a pas demandé ; pas de liste à puces. - Lecture : grep sur les symboles que le changement touche, puis lire diff --git a/README.md b/README.md index 7e2ddba..cbf1b4e 100644 --- a/README.md +++ b/README.md @@ -46,7 +46,7 @@ The full ruleset is in [The rules](#the-rules), under Reference. ## Measured -

Eval suite, nine cases, three runs each. Shorter, and more often right. Fable 5.2: 0.99 with crewcut, 0.88 for the same model with no plugin. Opus 5.5: 1.00 with, 0.85 without. Sonnet 5.5: 1.00 with, 0.86 without. Haiku 5.5: 0.98 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero.

+

Eval suite, nine cases, three runs each. Shorter, and more often right. Fable 5.2: 1.00 with crewcut, 0.88 for the same model with no plugin. Opus 5.5: 1.00 with, 0.84 without. Sonnet 5.5: 0.97 with, 0.86 without. Haiku 5.5: 0.97 with, 0.84 without. Score is the share of graders passed. Every case grades correctness as well as size: a shorter answer that is wrong scores zero.

Ponytail 5's benchmark, crewcut 0.14.0 against no plugin, Opus 5.5, 39 tasks, 5 runs. Lines of code minus 58 percent, 95 percent interval minus 68 to minus 47, better on 32 tasks and worse on 1. Code information minus 52 percent. Output tokens minus 56 percent. Cost minus 34 percent, interval minus 44 to minus 24. Time minus 53 percent. Turns minus 19 percent. Hidden checks: 85 of 90 passed, 86 without the plugin. Six security tasks: 30 of 30 in both arms. Privacy task: 0 of 5, then 8 of 10 after the rule that names what a change could expose. Ponytail 5 published for itself: code minus 53 percent, output tokens minus 45, cost minus 26, time minus 41. Two runs a month apart, intervals overlap.

@@ -71,10 +71,10 @@ session alone, without the judge: | Model | crewcut | Score with | without | Turns with | without | Cost per run with | without | |---|---|--:|--:|--:|--:|--:|--:| -| Fable 5.2 | 0.16.0 | 0.99 | 0.88 | 4.6 | 5.7 | 0.201 USD | 0.192 USD | -| Opus 5.5 | 0.16.0 | 1.00 | 0.85 | 5.0 | 5.4 | 0.093 USD | 0.087 USD | -| Sonnet 5.5 | 0.16.0 | 1.00 | 0.86 | 4.5 | 5.3 | 0.044 USD | 0.041 USD | -| Haiku 5.5 | 0.16.0 | 0.98 | 0.84 | 5.1 | 6.0 | 0.003 USD | 0.003 USD | +| Fable 5.2 | 0.17.0 | 1.00 | 0.88 | 4.6 | 5.7 | 0.203 USD | 0.192 USD | +| Opus 5.5 | 0.17.0 | 1.00 | 0.84 | 4.8 | 5.4 | 0.092 USD | 0.087 USD | +| Sonnet 5.5 | 0.17.0 | 0.97 | 0.86 | 4.3 | 5.3 | 0.043 USD | 0.041 USD | +| Haiku 5.5 | 0.17.0 | 0.97 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD | Ponytail 5's benchmark, Opus 5.5, 39 tasks, 5 runs, crewcut 0.14.0 against no plugin: @@ -247,7 +247,8 @@ What the ruleset says, beyond the ladder above. **Token discipline** - Output: no preamble, no recap, no unrequested explanation. Never paste - back code just written. Three sentences at most: what changed, where, and + back code just written. Three sentences at most: what changed, for a bug its + cause first, where, and one caveat only if it changes what you do next; no offer and no "skipped" line for what the ticket never asked; no bullet list. - Reading: grep for the symbols the change touches, then read only those diff --git a/assets/readme/evals-fr.png b/assets/readme/evals-fr.png index b39869b..1adc37c 100644 Binary files a/assets/readme/evals-fr.png and b/assets/readme/evals-fr.png differ diff --git a/assets/readme/evals.png b/assets/readme/evals.png index c6e195c..21b7eb0 100644 Binary files a/assets/readme/evals.png and b/assets/readme/evals.png differ diff --git a/assets/readme/src/evals-fr.html b/assets/readme/src/evals-fr.html index 324191f..c78fb68 100644 --- a/assets/readme/src/evals-fr.html +++ b/assets/readme/src/evals-fr.html @@ -11,10 +11,10 @@
Suite d'évals, neuf cas, trois runs chacun

Plus court, et plus souvent juste.

-
Fable 5.2
0,99
0,88
-
Opus 5.5
1,00
0,85
-
Sonnet 5.5
1,00
0,86
-
Haiku 5.5
0,98
0,84
+
Fable 5.2
1,00
0,88
+
Opus 5.5
1,00
0,84
+
Sonnet 5.5
0,97
0,86
+
Haiku 5.5
0,97
0,84
avec crewcutmême modèle, sans plugin
Le score est la part de graders réussis. Chaque cas note la justesse autant que la taille :
une réponse plus courte mais fausse vaut zéro.
diff --git a/assets/readme/src/evals.html b/assets/readme/src/evals.html index ffd455b..e7610aa 100644 --- a/assets/readme/src/evals.html +++ b/assets/readme/src/evals.html @@ -11,10 +11,10 @@
Eval suite, nine cases, three runs each

Shorter, and more often right.

-
Fable 5.2
0.99
0.88
-
Opus 5.5
1.00
0.85
-
Sonnet 5.5
1.00
0.86
-
Haiku 5.5
0.98
0.84
+
Fable 5.2
1.00
0.88
+
Opus 5.5
1.00
0.84
+
Sonnet 5.5
0.97
0.86
+
Haiku 5.5
0.97
0.84
with crewcutsame model, no plugin
Score is the share of graders passed. Every case grades correctness as well as size:
a shorter answer that is wrong scores zero.
diff --git a/evals/RESULTS.md b/evals/RESULTS.md index e33fae3..ad1dd0e 100644 --- a/evals/RESULTS.md +++ b/evals/RESULTS.md @@ -405,3 +405,65 @@ twice; Opus, `csv-export` `root-cause-stated` once; Haiku, five graders once each. 12.42 USD for this table, 7.09 USD for the three trials. Six Fable runs did not start (a process spawn error, twice that day already) and their two cases were run again. + +## The cause of a bug in the answer (crewcut 0.17.0, 2026-11-13) + +Three graders kept failing with the plugin on. Read side by side, the +failing and the passing answers gave two different stories. + +On `shared-bug` and `csv-export`, `root-cause-stated` failed when the +answer described the fix and left the cause to be inferred: "formatPrice +now converts numeric strings to numbers". The ruleset asked for "what +changed, where" and nothing more, and Fable, the model that holds the +three sentences most strictly, spent them on the fix and two caveats. The +Output line now reads "what changed (a bug: its cause first), where". + +| Plugin arm, `root-cause-stated` | Before | After | +|---|--:|--:| +| Fable 5.2, `shared-bug` | 1 / 3 | 5 / 5 | +| Opus 5.5, `csv-export` | 2 / 3 | 5 / 5 | + +On `date-picker`, every `short-answer` failure carried the same +sentence: the field was made required like the others, "drop required if +it should be optional". The passing answers made the same choice and said +it as a fact, or did not say it. The grader was written against offers of +a script or of a computed max date; naming a choice the ticket left open +and how to undo it is not one, so the grader now allows it. This is a +change of the measure, not of the plugin, and the arm without the plugin +was run again on that case with the new grader. + +The clause takes the ruleset to 634 estimated tokens at its heaviest +level; the ceiling goes from 630 to 640. + +The nine cases with the plugin, three runs, Opus as judge. The arm without +the plugin is the one of the earlier sections, `date-picker` excepted. + +| Model | Score with | was | without | Turns with | without | Session cost with | without | +|---|--:|--:|--:|--:|--:|--:|--:| +| Fable 5.2 | 1.00 | 0.99 | 0.88 | 4.6 | 5.7 | 0.203 USD | 0.192 USD | +| Opus 5.5 | 1.00 | 1.00 | 0.84 | 4.8 | 5.4 | 0.092 USD | 0.087 USD | +| Sonnet 5.5 | 0.97 | 1.00 | 0.86 | 4.3 | 5.3 | 0.043 USD | 0.041 USD | +| Haiku 5.5 | 0.97 | 0.98 | 0.84 | 5.2 | 6.0 | 0.003 USD | 0.003 USD | + +Fable and Opus pass every grader of every run. Sonnet and Haiku came out +lower than in the previous section, so both were run a second time, with +the new wording and with the one of 0.16.0: + +| Score with the plugin, nine cases, three runs | 0.16.0 wording | 0.17.0 wording | +|---|--:|--:| +| Sonnet 5.5, first suite | 1.00 | 0.97 | +| Sonnet 5.5, second suite | 0.97 | 0.98 | +| Haiku 5.5, first suite | 0.98 | 0.97 | +| Haiku 5.5, second suite | 0.99 | 0.99 | + +The same graders fail under both wordings: on Sonnet `explain-bug` +`full-explanation` and `url-parse` `short-answer`, twice each in a +suite, when the session reads one file less. The spread between two suites +of the same wording is as wide as the gap between the wordings: a suite of +three runs does not separate 0.97 from 1.00 on these two models, and the +1.00 published for Sonnet in 0.16.0 was the high end of that spread. Three +cases checked on five runs with the new wording: Sonnet `explain-bug` 4 +out of 5, Haiku `csv-export` 4 out of 5, Haiku `explain-bug` 5 out of 5. + +About 26 USD in all. Three Sonnet runs did not start in the first suite +(the same process spawn error) and their two cases were run again. diff --git a/evals/date-picker/case.yaml b/evals/date-picker/case.yaml index f076374..4969e7c 100644 --- a/evals/date-picker/case.yaml +++ b/evals/date-picker/case.yaml @@ -49,7 +49,8 @@ graders: bullet points. It names the file and the field. It has at most four sentences in all. No sentence offers or suggests a script, a computed or dynamic value, or another change to the form; saying that the server - still has to accept or check the field is fine. + still has to accept or check the field is fine, and so is saying that + the field was made required and that the attribute can be dropped. - type: regex name: plain-text-answer pattern: '[\u2013\u2014\u2018\u2019\u201C\u201D\u00A0\u202F\u2190-\u21FF\u2600-\u27BF\uD83C-\uD83E]' diff --git a/hooks/ruleset.md b/hooks/ruleset.md index dbdf940..95e0987 100644 --- a/hooks/ruleset.md +++ b/hooks/ruleset.md @@ -22,7 +22,7 @@ Comments: only a reason the code cannot show, one plain line; never a summary of Modern by default: follow the installed versions' current docs, compact forms when clear; unsure: read them; never a feature the version lacks. Tokens: -- Output: no preamble, recap or unrequested explanation; never paste back your code. Three sentences at most: what changed, where, one caveat only if it changes what the user does next; no offer or skipped line for what the ticket never asked; no bullets. +- Output: no preamble, recap or unrequested explanation; never paste back your code. Three sentences at most: what changed (a bug: its cause first), where, one caveat only if it changes what the user does next; no offer or skipped line for what the ticket never asked; no bullets. [lite] - Reading: prefer grep to reading whole files. [full] - Reading: grep what the change touches, then read only those files by line range; never read a file twice or to confirm what grep showed; no repo tour. [ultra] - Reading: grep first, then one read per file the change touches, by line range, never twice; no repo tour. diff --git a/skills/crewcut-gain/SKILL.md b/skills/crewcut-gain/SKILL.md index b0d0f01..8af4e99 100644 --- a/skills/crewcut-gain/SKILL.md +++ b/skills/crewcut-gain/SKILL.md @@ -24,10 +24,10 @@ crewcut gain measured on public repos and eval cases, not on t eval cases (nine, three runs each, same model with and without) score with / without turns with / without session cost with / without - Fable 5.2 0.99 / 0.88 4.6 / 5.7 -19 % 0.201 / 0.192 USD +5 % - Opus 5.5 1.00 / 0.85 5.0 / 5.4 -7 % 0.093 / 0.087 USD +7 % - Sonnet 5.5 1.00 / 0.86 4.5 / 5.3 -15 % 0.044 / 0.041 USD +7 % - Haiku 5.5 0.98 / 0.84 5.1 / 6.0 -15 % 0.003 / 0.003 USD 0 % + Fable 5.2 1.00 / 0.88 4.6 / 5.7 -19 % 0.203 / 0.192 USD +6 % + Opus 5.5 1.00 / 0.84 4.8 / 5.4 -11 % 0.092 / 0.087 USD +6 % + Sonnet 5.5 0.97 / 0.86 4.3 / 5.3 -19 % 0.043 / 0.041 USD +5 % + Haiku 5.5 0.97 / 0.84 5.2 / 6.0 -13 % 0.003 / 0.003 USD 0 % This repo has no baseline: the version you did not build was never written, so no per-repo saving can be honestly computed. diff --git a/skills/crewcut/SKILL.md b/skills/crewcut/SKILL.md index 2931f25..6582532 100644 --- a/skills/crewcut/SKILL.md +++ b/skills/crewcut/SKILL.md @@ -60,7 +60,7 @@ second ticket. No type alias or helper for a single use. ## Output Never paste back code you wrote. Reply in three sentences at most: what -changed and where, then one caveat, only if it changes what the user does +changed (for a bug, its cause first) and where, then one caveat, only if it changes what the user does next. `skipped: , add when: ` only for a corner the ticket needed and you cut; never for an option it did not ask for. No preamble, restating or recap. An explanation the user asked for is given in full; only diff --git a/tests/text.test.js b/tests/text.test.js index 96093d4..79aa5fb 100644 --- a/tests/text.test.js +++ b/tests/text.test.js @@ -66,12 +66,12 @@ test('commit messages use plain punctuation and carry no trailer', () => { assert.doesNotMatch(log, /^[A-Za-z-]+: .+$/m, 'trailer line found in a commit message'); }); -test('compact ruleset stays under 630 estimated tokens at every level', () => { +test('compact ruleset stays under 640 estimated tokens at every level', () => { const markdown = fs.readFileSync(path.join(root, 'hooks', 'ruleset.md'), 'utf8'); for (const level of LEVELS) { for (const markers of [false, true]) { const tokens = Math.ceil(renderRuleset(level, markdown, { markers }).length / 4); - assert.ok(tokens < 630, `${level}, markers ${markers}: about ${tokens} tokens`); + assert.ok(tokens < 640, `${level}, markers ${markers}: about ${tokens} tokens`); } } });