Conversation
Merge pull request #261 from slowdini/dev
feat(marketing): add skills-ref validation to CI and readme badges
Two prior investigating-bugs eval runs came back null (+0.00 on Sonnet 4.6, +3.7pp of noise on Haiku 4.5). The cause was not the skill but the cases: they had no attractive wrong answer, so both arms took the correct path and the delta measured nothing. Adds the failure-prevention vs quality-gradient distinction, the recipe for building a trap that actually baits (hard reproduction + a locally-rewarded wrong fix), and guidance for telling a real ceiling apart from an unbaited one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replaces the checkout-discount cases, whose wrong answer (clamp a negative amount to zero, visibly charging $0) no model was ever tempted by, with a date-only timezone bug that is hard to reproduce and has two locally-rewarded wrong fixes. - fixtures/tz-date-only: the bug reproduces only under a negative UTC offset, so the naive repro is green and "works fine for me" is misleading. - fixtures/tz-nudged: the same fixture with the seeded `+1 day` nudge already applied, so the seeded case's premise is in the code the agent opens. - Both cases now grade against the held-out TZ matrix via command_check. Every tempting fix fails at least one cell; only treating the value as a timezone-agnostic calendar date passes all four. - Strips hints from the fixture comments — repro.ts previously spelled out that reproducing meant varying TZ, which handed the agent both TZ assertions and would have ceiled them in either arm. - Drops the per-eval `runs` overrides so a single --runs applies uniformly. Adds a tracked harness descriptor that pins dispatches to TZ=UTC (the suite is inert on a host in a negative-offset zone) and drops user-scope settings so the control arm is genuinely skill-free. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A pilot dispatch showed the built-in claude-code recipe's --permission-mode acceptEdits auto-approves edits but not Bash; with stdin detached every non-trivial command is auto-denied. `ls`/`grep`/`find` ran, `bun run repro.ts` and `node -e` did not, and the agent finished by stating it could not execute the reproduction and had reasoned statically instead. That is fatal for this suite specifically: the behaviour under measurement is "reproduce by varying the timezone before fixing", so with execution blocked neither arm can reproduce anything and the transcript check passes on an attempt that never ran. Raises the dispatch permission mode in the descriptor and documents why. The write guard remains the sandbox boundary and still enforces — verified in the same probe that ran bun successfully. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…fication case Two changes from reading the pilot transcripts. The timezone judges enumerated techniques rather than testing the property. The unskilled arm parsed to local midnight AND fixed serializeForSave to match, which is correct in every timezone cell — but the rubric blanket-failed local parses, so it would have scored a correct answer as FAIL. Both rubrics now judge whether every consumer is consistent with the representation the agent chose. The pilot also showed a real behavioural difference the suite was blind to: the with_skill arm left regression tests behind in both cases (tz.test.ts, repro-overdue.ts, repro.ts +40 lines), the without_skill arm left none. That is Phase 4 skill doctrine, so it gets its own case rather than being bolted onto the timezone ones. pagination-drops-last-page-cold grades it by mutation: run everything the agent left behind against its own fix (must be green), then restore the original bug and re-run (something must now fail). Verified against five hand-built final states — no test, real test, vacuous test, unfixed, and repro script — before first use. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pagination case ceiled at 3/3 in both arms on Sonnet 5 and again on Haiku. Cause: it shipped chunk.fixture.ts, so every unskilled run appended a correct regression test to the file already sitting there. Handing the agent an obvious place to put a test removes the decision the case exists to measure. Deleting all tests would overcorrect — in a project with no tests, declining to write one is a defensible reading of local convention. So the fixture now ships formatBytes.ts WITH its tests, and chunk with none: the project visibly tests its code, the buggy function is visibly untested. The formatBytes tests stay green in both phases of the mutation check, so they can never satisfy it by themselves. Re-verified all five discriminator scenarios against the new layout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Run B's +11.9pp aggregate delta comes entirely from two of four cases; the other two contribute zero signal and full variance. That caps what any affordable n can show, so fix the predictions, analysis plan and decision rules now rather than after reading results. Declared in advance: the aggregate pass-rate is demoted to a secondary endpoint and stays underpowered at n=10 (1.4 SE) — reaching 80% power would need ~320 dispatches. Efficiency (tokens, wall clock) becomes the primary endpoint at ~5.4 SE, with two assertion-level discriminators co-primary. The suite is frozen here: no case dropped, no fixture revised after results. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--agent-model was omitted, so all 80 dispatches ran on Sonnet 5 rather than the pre-registered Haiku 4.5. Append a deviation record rather than editing the frozen predictions. The pre-registered decision rule returns DO NOT PROMOTE: neither co-primary reached significance (P2 ceiled 10/10 vs 10/10, P3 p=0.47) and the validity gate failed (invocation 39/40). Two findings survive regardless of population. P1 hit its pre-specified numbers (10/10 vs 5/10, p=0.0325), verified not to be a guard artifact -- three runs never wrote verification, two wrote it and deleted it. And P5 is refuted in reverse: +24.8% tokens on Sonnet 5, not -41%, so the token saving was unskilled-Haiku flailing rather than a property of the skill. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
P1 hit its pre-specified numbers in the discovery run but on an unplanned population. Test it again on a declared one, with P1 as the stated primary rather than one of seven predictions. Prediction is set deliberately looser than the discovery point estimate (delta >= +0.25 rather than the observed +0.50) so the test can fail. The failure branch is declared too: report it unreplicated, do not re-tune the fixture, do not raise n and retry. Models are passed explicitly and verified in a smoke dispatch before the fleet -- omitting them falls through to the session default with no error, which is what corrupted the previous run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The P1 replication cleared every pre-registered threshold on a declared model: left_a_regression_test 20/20 vs 9/20, p=0.000145 (prediction was delta >= +0.25, p < 0.05). Control ceiled as designed, invocation 20/20, zero live-source reads, zero write violations. All 33 guard denials were Bash-redirect false positives -- no Write was ever blocked, so the guard cannot explain a missing test file. Pooled with discovery: 30/30 vs 14/30, p = 1.9e-06. The failure mode is not the obvious one. Of 11 failing without_skill runs, 7 wrote verification in-sandbox, ran it, and then deleted it; only 4 never wrote one. The skill changes durability, not diligence. Reframe the README table around this. A single "improvement" column assumes every skill's value is one pass/fail outcome; investigating-bugs barely moves whether Sonnet 5 fixes the bug, because it already does. Name the behavior instead, and report the +58% token cost next to the gain rather than burying it. Record what ceiled, too: both timezone cases are flat in both arms on this tier. They stay in the suite -- dropping a case once it stops flattering the skill is how a suite gets tuned into agreement -- but they are diagnostic, not evidence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…mposition The investigating-bugs campaign showed that Mode A suites authored skill-at-once measure whichever behavior they accidentally trip over: the suite's only confirmed effect (durable verification, p=1.9e-06 pooled) maps to a single instruction and was discovered by accident, while 3 of 4 cases ceiled because their target behaviors are base-Sonnet-5 native. Reframe: a Mode A eval is a bundle of per-behavior removal tests. New "Decompose the skill into behaviors first" subsection (inventory the skill as functional units - a behavior plus every lexical restatement of it - and map cases to behaviors both directions), and a "Pricing a behavior - ablation runs" section (Mode B with the expectation inverted, delete the whole functional unit, null ablations are per-tier verdicts, runs are targeted while the map is mandatory). The 3ff7054 additions are absorbed under this framing: trap-engineering is the per-case half (what makes a case sensitive to its behavior), and a real ceiling is now read as a null ablation recorded in the coverage map rather than a verdict on the skill. Iron Law: declare-and-skip. This is authoring-craft guidance for a manually-invoked meta-skill; its validation is applying the process to each skill's suite one-by-one, starting with investigating-bugs (COVERAGE.md in the next commit). Door stays open for a dogfood eval if wanted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First application of the decomposition process from evaluating-skills, as its worked example. One entry per functional unit (behavior + every lexical restatement), mapped to its covering case and measured status from the promoted baseline: B1 durable-verification confirmed (+55pp, carries the baseline), B2/B3 ceiled on Sonnet 5, B4 flaky-diagnosis not significant, and eight contingent behaviors explicitly uncovered (Phases 1.1, 1.3, 1.4, 2, 3, the three-fix limit, no-bundled-refactoring, and B-verify) - each a candidate future case, none a claim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first ablation run under the new process, frozen before dispatch: Mode B with old=full skill, new=skill minus B1's entire functional unit (Phase 4.1, the TDD REQUIRED BACKGROUND line, the verification red-flag bullet, with Phase 4.3 minimally rewritten so B-verify survives without a dangling reference). pagination-drops-last-page-cold only, n=10/arm, Sonnet 5. Prediction: the ablated arm collapses toward the no-skill rate (<=0.65 vs >=0.85, p<0.05). Explicit failure branch: record the effect as diffuse and revise the coverage map - no wider deletion, no retry, no fixture change. Not yet dispatched; the run waits on an operator-confirmed pre-flight summary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…CONFIRMED The pre-registered prediction held on every threshold: full skill 10/10 vs B1-ablated 5/10 on left_a_regression_test (delta +0.50, Fisher p = 0.0325, n=10/arm, Sonnet 5), control ceiled at 10/10 both arms, validity clean (invocation 1.0/1.0 both arms, live_source_reads 0, model id verified in every events file, no guard denial causally linked to a primary failure). Deleting the durable-verification behavior's three lexical expressions reverted the measured behavior to the no-skill discovery rate (5/10) while the rest of the skill stayed loaded and invoked. The coverage map's case-to-behavior link for B1 is now validated by ablation, not assumed - the first confirmed data point for the decomposition process. Observations recorded, not claimed (n=10): ablated failures split 1 wrote-then-deleted / 4 never-wrote (all verified inline instead); the ablated arm's five passing runs were its five most expensive (881k-1.48M tokens vs the full arm's tight 430-596k). Baseline untouched, fixture untouched, live SKILL.md untouched, per the frozen decision rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-ablation investigating-bugs eval baseline + behavior-decomposition framing for evaluating-skills
Ship slow-powers as a Cline plugin (CLI/SDK/Kanban) and set up Cline tooling for working on this repo. - cline/plugins/slow-powers.js: registers bootstrap.md as a session rule (registerRule) and gates the first switch_to_act_mode of each conversation on hardening-plans (skip-once + per-conversation marker, mirroring hooks/exit-plan-mode's deny-once argument) - package.json: cline.plugins manifest field; cline/ added to files. Skills are auto-discovered from the package root, no wiring needed. - tests: Cline entry in tests/harness/spec.ts, manifest assertions in manifests.test.ts, and cline-plugin.test.ts covering plugin shape and gate behavior - docs: README Cline install section with the IDE-extension caveat; AGENTS.md four-harness update + live-testing instructions - repo setup: .clinerules/memory-bank.md + seeded memory-bank/; .gitignore excludes .cline/plugins/ install artifacts No skill content changed, so no evals are required.
feat: add Cline harness support
slowdini
added a commit
that referenced
this pull request
Aug 8, 2026
Merge pull request #267 from slowdini/dev
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release notes